iO dZddlmZddlZddlmZmZddlmZddl m Z m Z m Z m Z ddlZ ddlZddlmZdZeGddZeGd d ZGd d Zd*d Zd+dZd,dZedk(rej>dZ d- d.dZ!ee e!ddde!ddde!ddde!ddde!ddd g!Z"e"jGd"d d#Z$e%d$e%d%e$jLjOd&e$jPd'e$jRd(e%d)e$jLjUe jUz jVyy#e$rZdZdZeZYdZ[z-OptionChain.find_target_leg..}s#bejFXBYr"keyz-No contracts with finite delta at expiration u (DTE≈z , target DTE=z).cXtt|jtz S)N)r-rr)c target_deltas r#r0z-OptionChain.find_target_leg..sU177^eL>Q-Q)Rr")r( ValueError_normalize_option_typerr&rpd Timestamp normalizerintdaysminitemsmathisfiniter)selfr/r5rwantr4typedr&expiriesexpdtebest_expbest_dtebuckets `` r#find_target_legzOptionChain.find_target_legTs$~~LM M%k2 NNDqammt.CDD=dXT$**//J[I\\]^_ _ TZZ(224-/ A,,q||,668CsU{(()CQwHSM  XY Y!!17YZ( ||ALL)335A agg&   ? ?PQ"= B@  6RSSEE* sGG2AGN)r/r;r5rrrreturnr ) rrrrr rlistr(rJr!r"r#r%r%HsO &+D&AI#A8T8T8T 8T  8Tr"r%cLeZdZUdZiddddddddddddd d d d d d d d ddddddddddddddZded<gdZded<d%dZedddd d&dZ ed'd Z edddd d(d!Z e d)d"Z d*d#Z d+d$Zy),IVolatilityLoadera Load iVolatility CSV (via Parquet cache) and query single-day chains. ``COLUMN_MAP`` maps **raw CSV header** -> **internal column name** used in Parquet and in memory. Adjust keys to match your export (e.g. some files use ``implied_volatility`` instead of ``iv``). rrrzcall/putrcall_putzCall/Putimplied_volatilityrIVrrpricez mean pricerrrrrhozClassVar[dict[str, str]] COLUMN_MAP rrrrrrrrrrrrzClassVar[list[str]] _PARQUET_COLSct|j|_|jjj dvryy)z Parameters ---------- file_path Path to a **Parquet** file produced by ``convert_to_parquet``. Using Parquet keeps random access and row-group filters fast and RAM low. >.pq.parquetN)r expanduser file_pathsuffixlower)rAr[s r#__init__zIVolatilityLoader.__init__s=i335 >> & & (0C C  Dr"i@ )rrN) chunksize date_columns float32_colsctttdtt |j }t |j }|j jdd|d}tj||}d} |D]}tj|tj||}|jr7tjj|d } |)tj t#|| j$}|j'|  ||j)|S#||j)wwxYw) a Stream-read a large CSV in chunks, normalize columns, downcast floats, write Parquet. * Dates are parsed with ``pd.to_datetime`` (UTC-naive midnight normalized). * Greeks, IV, bid/ask/mid/strike use ``float32`` to halve RAM vs float64. * ``mid`` is ``(bid+ask)/2`` when not otherwise supplied from ``price``. Parameters ---------- chunksize Rows per read from CSV; tune based on available RAM and column count. NzFconvert_to_parquet requires pyarrow. Install with: pip install pyarrowTparentsexist_ok rrrrrrrrrr_ column_mapr`raFpreserve_index)pqpa ImportError_PYARROW_IMPORT_ERRORrrZparentmkdirr8read_csvrN_standardize_chunkrTemptyTable from_pandas ParquetWriterrschema write_tableclose) csv_path parquet_pathr_r`rareaderwriter raw_chunkstdtables r#convert_to_parquetz$IVolatilityLoader.convert_to_parquetsA* :X( )>,,.L)446 !!$!>   LX;*. # * '::0;;!-!- ; 99,,S,G>--c,.?NF""5) *! ! "s BD77E cxt|j}t|jdt}|S)u Return iVolatility split exports in a stable order: ``data_download.csv`` first, then ``data_download 2.csv`` … ``data_download 7.csv`` by numeric suffix. Use this when combining multiple shards from the same vendor export. zdata_download*.csvr1)rrZsortedglob_ivolatility_csv_sort_key) directorydpathss r#sorted_ivolatility_csv_pathsz.IVolatilityLoader.sorted_ivolatility_csv_pathss2 O & & (qvv239RS r"c tttdtt |j }|j jdd|d}d}d} |D]}t |j }|jstd|tj||} | D]} tj| tj|| } | jr7tj j#| d } |)tj$t'|| j(}|j+| d} ||j- |s t/d |S#||j-wwxYw) u' Merge several iVolatility CSV shards into **one** Parquet file (streaming). Each file is read in ``chunksize`` rows so memory stays bounded. Schema is unified via ``COLUMN_MAP`` (e.g. ``mean price`` → ``price`` for files that omit ``is_settlement``). NzKcombine_csvs_to_parquet requires pyarrow. Install with: pip install pyarrowTrcrfFzCSV not found: rgrhrjz@No rows written from csv_paths (empty files or schema mismatch).)rlrmrnrorrZrprqis_fileFileNotFoundErrorr8rrrNrsrTrtrurvrwrrxryrzr6) csv_pathsr|r_r`rar~ wrote_anyr{pathr}rrrs r#combine_csvs_to_parquetz)IVolatilityLoader.combine_csvs_to_parquets :]( )L)446 !!$!>   L+/  % %H~002||~+odV,DEETY?!' %I+>>!#4#?#?%1%1 ?C yy HH00U0KE~!#!1!1#l2CU\\!R&&u- $I %  %(! _` ` ! "s C/E..Fc|jt|}|D]F}||jvstj||dj j ||<Hd|jvr|djt|d<d|jvrd|jvrtj|dd|d<nqtj|jddtj|jddzd z |d<ntj|dd|d<d D].}||jvstj||d||<0gd }|j|Dcgc]}||jvs|c} }|jr|S|D](}||jvs||jd ||<*tjD]8}||jvs|dvrtj nt"j$||<:|tjj'}|Scc}w)zLApply renames, parse dates, compute mid, coerce dtypes, keep Parquet schema.columnscoerceerrorsrrrRrrg@)rrrrrrrr)rrrrrr)subsetfloat32)rrr)renamedictrr8 to_datetimedtr:mapr7 to_numericgetdropnartastyperNrVNAr?nancopy) dfrir`rachunkcolr4needouts r#rsz$IVolatilityLoader._standardize_chunkbs0 $z"2 3 XCemm#^^E#JxHKKUUWc  X EMM )#(#7#;#;x PSUS`S`afajajkpaq{CTD!DHK Ke ==uhGE%LR DAEMM!==q(Ca D M $K1U]]8JQ$K L ;;L 6AEMM! 8??95a 6 #00 ]A %$%)N$N255TXT\T\a ]%33499; %Ls I I c*ttdttj|j}t }t |jD]}|j|dgjd}tj|jdjj}|jj!D]&}|j#tj$|(t't)|S)zBYield unique ``date`` values present in the Parquet file (sorted).zpyarrow is requiredrrrrr)rlrnro ParquetFiler[setrangenum_row_groupsread_row_groupcolumnr8r to_pandasrr:runiqueaddr9iterr)rApfdatesirsus r#iter_chain_datesz"IVolatilityLoader.iter_chain_datess :34:O O ^^DNN +#&5r(() +A##Ax#8??BCs}}x@CCMMOAXXZ&&( + ",,q/* +  + F5M""r"c0ttdttj|j }dd|fgdd|j fgdd|jfgg}d}tj}|D]3} tj|j|j|d}d}n||jr td||jr t|gStj |dd j"j |d<||d|k(}|jr t|gS|j%D cgc]\}} t'| } }} t|| S#t$r }|}Yd}~ d}~wwxYwcc} }w) a Load **only** rows for ``target_date`` using Parquet predicate pushdown. Uses the PyArrow engine so filters prune row groups where possible. If your ``date`` column was written as plain ``date32``/``timestamp`` without timezone, equality against a normalized ``pd.Timestamp`` is typically recognized by the scanner. We do **not** fall back to a full-file read (that would defeat the purpose on multi-GB datasets). Nz#get_chain_for_date requires pyarrowrz==pyarrow)rfiltersenginezCould not apply Parquet row filter on 'date'. Re-run convert_to_parquet so 'date' is timezone-naive datetime64, or pass a filter literal matching the on-disk type.r&r(rr)rlrnror8r9r: to_pydatetimer DataFrame read_parquetr[rV Exceptionrt RuntimeErrorr%rriterrows_row_to_contract) rA target_datetsfilter_variantslast_errrre_rr(s r#get_chain_for_datez$IVolatilityLoader.get_chain_for_dates :CDJ_ _ \\+ & 0 0 2dB  dB,,. / 0dBGGI & '=  &* \\^& G __NN ..#$      BHHF   88R26 6^^BvJx@CCMMO6 6 b ! 88R26 657[[]CTQ%a(C Cy99)  &Ds/E9F9 FF  F)r[ str | PathrKNone) r{rr|rr_r;r`tuple[str, ...]ratuple[str, ...] | NonerKr)rrrKz list[Path]) rzSequence[str | Path]r|rr_r;r`rrarrKr) r pd.DataFramerizdict[str, str]r`rrarrKr)rKzIterator[pd.Timestamp])rr rKr%)rrrrrTr rVr^ staticmethodrrrrsrrr!r"r#rNrNs ,,l, (, M , M , M, d, d, u, u, , g, , , !," #,$ u%,J(, *M&   !(>/3 ?? ? ? & ? - ? ??B   !(>/3 A'A A A & A - A AAF. .#.& . & .  ..` #6:r"rNc|j}|dk(ryd}|j|r+|t|dj} dt |dfSdd|fS#t $rdd|fcYSwxYw)uP``data_download.csv`` first, then ``data_download 2.csv`` … by numeric suffix. data_download)rrzdata_download Nrrr,)stem startswithlenstripr;r6)prprefixsufs r#rrs 66D  F v3v;=!'') s3x$ $ q$< q$<  s AA'&A'c|%t|trtj|ryt |j j }|jdry|jdry|dvr |dk(rdSdS|r|ddSdS)NrCP>PUTCALLrr,) isinstancerr?isnanrrupperr)xrs r#r7r7syZ5)djjm A A||C||CO6ks*s*1Ra52r"cttj|djtj|djt |dt |dt |dt |dt |dt |dt |d t |d t |d t |d  S)NrrrrrrrrrrrrrU)r r8r9r:rr)rs r#rrs  \\!F) $ . . 0<<,0::<Qx[!-() !E(O !E(O !E(O 4>AgJAgJAgJ 1V9   r"__main__z 2024-06-15rcbtj|}tt|||dddd|ddd S) Ng?g?g?g?g{Gz?g{GzgQ?rU)r8r9r day)rErroptexp_tss r#mkrsC c"  r"z 2024-07-10g@@gz 2024-07-17g @g ףp= g~@gffffffֿg`@g)\(z 2024-08-15g~@gr)r/r5rz,Demo: 30 DTE (closest bucket), ~10-delta putz Selected: exp=z strike=z delta=z.3fz DTE=)rrrKztuple[int, int, str])rrrKr)rz pd.SeriesrKr )r) rErrrrrrrrKr ),r __future__rr? dataclassesrrpathlibrtypingrrr r pandasr8rrmpyarrow.parquetparquetrlrornrr r%rNrr7rrr9rrchainrJlegprintrrrrr:r<r!r"r#rs0# (44! !  * CTCT CTVC:C:V  * z ",,| $C          0  |UE * |UE * |UE * |UE * |UE *   E   2Es  SC 89 S^^00238CJJ