Feeds:
Posts
Comments

Archive for April, 2017

Inbox
x

zulfikar.ahmed@gmail.com <zulfikar.ahmed@gmail.com>

Attachments8:31 PM (14 hours ago)

to harrington, jharris, jhp, jhricko_4, jianjunp, jinha, jjbrehm, jlind, jlondon, jlw, jmateo, jmerseth, jmetcalf, jmg, jmogel, joel, joelms, john.aldrich, john.beatty, johncrawford53, jose.oliveira, josesoto, josue, jpadgett, jpbalz
 Ladies and Gentlemen,
Today I would like to announce a concrete measurement of free lunch in US Equities.
The fundamental significance is that:

(a)  ARBITRAGE OPPORTUNITIES ARE NOT ANOMALOUS BUT UNIVERSAL

(b) OUR FUNDAMENTAL MODELS OF THE MARKETS ARE MISSING KEY FEATURES
As George Soros keeps repeating, there is no equilibrium in the market.  The lack of equilibrium in the market is sufficiently bad that arbitrage is UNIVERSAL.  The idea that arbitrage removes the profit opportunities is false.  And the work below is not particularly deep or complicated; it may be true that deep and complicated methods can yield profitable strategies but the importance of the following is that in fact SIMPLE EASY strategies can do this as well and the latter are more important to understand how we don’t understand the financial markets at all and all our fundamental theories are sort of missing glaring large problems.

Recall that finance theory has a fundamental pillar:  the Fundamental Theorem of Asset Pricing, the equivalence of No Free Lunch and Martingales.  All of option pricing theory is built on this pillar.  Option values are well-defined only because the underlying return process can be assumed to be a MARTINGALE.  (This is well-known from Harrison-Pliska late 1970s to Delbaen-Schachermayer mid 1990s).
So this is a beautiful academic theory which is almost universally wrong in equities markets.  How to see this?  Very simple.  An AR(1) or AR(2) process is not going to be a martingale generally: if x_t = b*x_{t-1} + noise, then obviously E[x_t] = b*E[x_{t-1}] with white noise and unless b=1 we don’t have a martingale; the same with AR(2).

So the fundamental empirical issue is whether returns are AR(1) or AR(2) with any statistical significance.  For this, grab daily returns from Jan-2007 to Apr-2017 and record the autocorrelations at lags 1 and 2.  The results are that in 403 stocks 68.98% have either lag 1 or lag 2 autocorrelations significantly greater than the 95% confidence interval.

Here is an actual list of tickers with ‘**’ marked next to the tickers with autocorrelation that crosses the significance threshold.  The interpretation is that the financial theory of no arbitrage models are not good models of the markets.  That the theoretical models do not match the markets is not so interesting.  What is interesting is that this is true with strong enough statistical significance for 70% of a random pool of stocks (these stocks are the ones used by sector indices so they are liquid large companies).
Before considering the data, let’s emphasize that this is not a peripheral issue: the fundamental models of the markets and the fundamental ways in which we think about the behaviour of the markets seriously (Black-Scholes and other option pricing etc) are wrong not in anomalous situations but almost UNIVERSALLY.  Now you could say, well quant hedge funds and technical traders know all this.  The problem is that this is not only an issue of making profits; it’s an issue for the world because how will we have any regulation of stable markets etc. if we PROFESSIONALISE a theory of markets that is just completely not matching empirical behaviour?
As for reproducing these results, rather trivial using some of the functions in my code.
 
 
symbols<-c(XLESymbols,XLFSymbols,XLVSymbols,XLFSymbols,XLBSymbols,XLISymbols,XLESymbols,XLUSymbols,XLYSymbols,XLKSymbols)
 
R<-constructComponentRets(symbols,as.Date(‘2007-01-01’))
R[is.na(R)]<-0
 
 
AllStocks<-matrix(0,403,2) # to hold 2 lag autocorrelations
for (k in 1:403){
AllStocks[k]<-acf(coredata(R[,k]))$acf[2:3]
}
 
for (k in 1:403){ ci<-qnorm((1+0.95)/2)/sqrt(length(index(R)));x<-AllStocksAcf[k,];condition<-(abs(x[1])>ci || abs(x[2])>ci);if(condition){c<-c+1}; disp(paste(symbols[k],x[1],x[2],ifelse(condition,’**’,”),ci))}
 
HES -0.0382046321294291 -0.0570669865098722 ** 0.0377825479458521 
CVX -0.0997064687533108 -0.0573128819839226 ** 0.0377825479458521 
COP -0.0456617159454092 -0.0586681694691485 ** 0.0377825479458521 
OXY -0.0817297336229358 -0.0674534813906045 ** 0.0377825479458521 
XOM -0.154390931073284 -0.0937170896373303 ** 0.0377825479458521 
TOT -0.0410477829041835 -0.0726442007287631 ** 0.0377825479458521 
APC -0.0191200093891514 -0.0486279651769037 ** 0.0377825479458521 
RDS-A -0.0492678418517147 -0.0694310181399589 ** 0.0377825479458521 
BP -0.0373603425230508 -0.063366917972029 ** 0.0377825479458521 
MRO -0.0290751816452744 -0.0529267122907142 ** 0.0377825479458521 
MPC 0.017306189764944 0.0449958365717394 ** 0.0377825479458521 
VLO -0.00345941128970282 -0.0602516199610069 ** 0.0377825479458521 
SUN 0.0852300471426968 -0.00491511937371354 ** 0.0377825479458521 
EC -0.0869753722946993 0.0287433678424457 ** 0.0377825479458521 
NBL -0.0502316224336053 -0.0471790215190326 ** 0.0377825479458521 
PBR 0.00342045429725232 -0.0508831766577323 ** 0.0377825479458521 
PSX 0.0692399362629805 0.0115581213823195 ** 0.0377825479458521 
PTR -0.0582862510082401 -0.0126595761879041 ** 0.0377825479458521 
STO -0.0731744314134723 -0.0378427683283295 ** 0.0377825479458521 
SU 0.00296420149908658 -0.0204177001582495  0.0377825479458521 
XOI -0.0825226884489279 -0.0714079178125618 ** 0.0377825479458521 
USO -0.0504181271525021 0.00401645415989441 ** 0.0377825479458521 
UWTI -0.011906902707701 0.0027545420837071  0.0377825479458521 
UCO -0.00928451932284817 0.0112879344467586  0.0377825479458521 
XLE -0.0882425761390582 -0.0674070863899576 ** 0.0377825479458521 
OIL -0.0355705914571595 0.0190243100411833  0.0377825479458521 
DWTI -0.0554859100583393 0.023015554550779 ** 0.0377825479458521 
SCO 0.00853833435477296 -0.0258144935527226  0.0377825479458521 
DBO -0.0512506061901174 0.00897109753967469 ** 0.0377825479458521 
VDE -0.0876993157109045 -0.0545651107028085 ** 0.0377825479458521 
DIG -0.0879856196103714 -0.0604525070847256 ** 0.0377825479458521 
BRK-B -0.0181581035348482 0.00552981244741128  0.0377825479458521 
JPM -0.112168793611472 0.00382949994664429 ** 0.0377825479458521 
WFC -0.123560698020111 0.0265080557183556 ** 0.0377825479458521 
BAC -0.0375718266905737 0.0755510549179119 ** 0.0377825479458521 
C 0.00215537806265875 -0.00117259389287589  0.0377825479458521 
GS -0.0488775328831961 -0.0135121943728409 ** 0.0377825479458521 
USB -0.107830473836386 0.0341711076492987 ** 0.0377825479458521 
CB -0.0901844998495054 -0.0985386278446791 ** 0.0377825479458521 
MS 0.00227834951905075 -0.0920449700517052 ** 0.0377825479458521 
AXP -0.104327766762045 -0.0162314701226256 ** 0.0377825479458521 
ICE -0.0811525508138042 -0.0378088620567959 ** 0.0377825479458521 
BBT -0.136991907923339 0.0280298699401075 ** 0.0377825479458521 
TRV -0.20168222412595 -0.0437715964556488 ** 0.0377825479458521 
SPGI -0.00645881348352848 -0.0335211126805885  0.0377825479458521 
AON -0.113506441431514 -0.0262821934708995 ** 0.0377825479458521 
AFL -0.203097367224819 0.10711590165834 ** 0.0377825479458521 
ALL -0.109974287818419 -0.0204586085995221 ** 0.0377825479458521 
STT -0.146763172622148 -0.0299751300500908 ** 0.0377825479458521 
STI -0.0612653830209914 0.0213140855465753 ** 0.0377825479458521 
DFS -0.0836161625072747 -0.00547529490921607 ** 0.0377825479458521 
RF -0.00810885842382611 -0.073715591613829 ** 0.0377825479458521 
PFG -0.0477039562119986 -0.0497092833824634 ** 0.0377825479458521 
TROW -0.123655292200166 -0.0457198993648533 ** 0.0377825479458521 
WLTW -0.0372185135152648 -0.0824654326904595 ** 0.0377825479458521 
LNC -0.0462211872989022 -0.00166199148230785 ** 0.0377825479458521 
BEN -0.0523441200109036 -0.0424716019423321 ** 0.0377825479458521 
HBAN -0.082539241511326 0.0504037793277749 ** 0.0377825479458521 
L -0.15931989931682 0.00298517418800225 ** 0.0377825479458521 
IVZ -0.113467825133864 -0.00714649451463555 ** 0.0377825479458521 
CMA -0.0724396726661677 -0.00314028752852232 ** 0.0377825479458521 
CINF -0.174034521842851 -0.0751836604271167 ** 0.0377825479458521 
UNM -0.156895756787395 -0.0334000841127673 ** 0.0377825479458521 
XL -0.0171271951898044 -0.0163243178916789  0.0377825479458521 
AJG -0.0910118583535343 0.00217679533424188 ** 0.0377825479458521 
ETFC -0.00508780255018044 -0.00809512138027311  0.0377825479458521 
RJF -0.0797183553944291 -0.0157614629957661 ** 0.0377825479458521 
AMG -0.0563830729562557 0.0341842130634544 ** 0.0377825479458521 
LUK -0.047614023063904 0.00217716302892084 ** 0.0377825479458521 
TMK -0.111065929777258 0.0326710381984407 ** 0.0377825479458521 
ZION -0.0281758159340018 0.00889686303588805  0.0377825479458521 
NDAQ -0.0338723081842863 0.00430166275574087  0.0377825479458521 
CBOE -0.0532537993511472 0.066195481651599 ** 0.0377825479458521 
PBCT -0.130790709218224 -0.0454650521456619 ** 0.0377825479458521 
AIZ -0.0949937122321678 0.000423970731246187 ** 0.0377825479458521 
NAVI 0.0547632105906803 0.0330805652408505 ** 0.0377825479458521 
JNJ -0.0705479102171742 -0.0749617669508772 ** 0.0377825479458521 
PFE -0.0604774784203281 -0.0646646879796013 ** 0.0377825479458521 
MRK -0.0360385149769978 -0.0423564396973475 ** 0.0377825479458521 
UNH -0.0322104907641962 -0.018225202635997  0.0377825479458521 
AMGN -0.0619744906445128 -0.0276969578720048 ** 0.0377825479458521 
MDT -0.0282306594652347 -0.00794701675692433  0.0377825479458521 
ABBV -0.0202318682760407 -0.0301784190704243  0.0377825479458521 
CELG -0.0761989024754419 -0.0171393343629497 ** 0.0377825479458521 
BMY -0.0349873636658062 -0.047243117143378 ** 0.0377825479458521 
GILD -0.043055382010135 -0.0319339234783084 ** 0.0377825479458521 
AGN 0.00800492609498722 -0.0634554591660675 ** 0.0377825479458521 
LLY -0.0925796969205498 -0.0610658275521083 ** 0.0377825479458521 
ABT -0.0436711636308865 -0.0367701499610535 ** 0.0377825479458521 
TMO -0.045253239357858 -0.0742932176438085 ** 0.0377825479458521 
BIIB -0.0516766500998773 -0.0414792434760039 ** 0.0377825479458521 
DHR -0.0509380626989889 -0.0551173706319112 ** 0.0377825479458521 
AET -0.0516364732727262 -0.0410965909283735 ** 0.0377825479458521 
ANTM -0.00403706396644323 -0.0326105923903629  0.0377825479458521 
ESRX -0.0304436837274782 -0.00411197397349527  0.0377825479458521 
SYK -0.00850273679748132 -0.0403472314022265 ** 0.0377825479458521 
CAN -0.000376070668059349 -0.000374494309213882  0.0377825479458521 
ZTS -0.0999795753546863 -0.0391137763119649 ** 0.0377825479458521 
ILMN 0.0188985722345515 -0.0192753169943724  0.0377825479458521 
HCA 0.0234777239235045 -0.0115960909011663  0.0377825479458521 
ZBH 0.0152673975591984 -0.0571768089057087 ** 0.0377825479458521 
INCY -0.021587829604441 0.0258937453428128  0.0377825479458521 
EW -0.00571564799539204 -0.0286290905299879  0.0377825479458521 
BCR -0.0595965218370414 -0.022544048861204 ** 0.0377825479458521 
MYL 0.0374537404777165 -0.00667307687191832  0.0377825479458521 
CERN -0.0147894932726164 -0.0114835302768089  0.0377825479458521 
HOLX -0.0126528588020796 -0.0453922469684015 ** 0.0377825479458521 
UHS -0.0234136471536489 -0.000964697108169621  0.0377825479458521 
DVA -0.0113621430834923 -0.0671767361318249 ** 0.0377825479458521 
PRGO 0.0163485595002214 -0.0272421168197988  0.0377825479458521 
COO 0.00847911115321135 -0.0272223547558006  0.0377825479458521 
VAR -0.0382548299079038 -0.0297145014608856 ** 0.0377825479458521 
EVHC 0.00721958058009644 -0.00200833374727628  0.0377825479458521 
PKI -0.0452878276107661 -0.0475665789583021 ** 0.0377825479458521 
MNK 0.0384322491144136 -0.0220263430589388 ** 0.0377825479458521 
PDCO -0.0574646021985127 -0.0186511743902512 ** 0.0377825479458521 
BRK-B -0.0181581035348482 0.00552981244741128  0.0377825479458521 
JPM -0.112168793611472 0.00382949994664429 ** 0.0377825479458521 
WFC -0.123560698020111 0.0265080557183556 ** 0.0377825479458521 
BAC -0.0375718266905737 0.0755510549179119 ** 0.0377825479458521 
C 0.00215537806265875 -0.00117259389287589  0.0377825479458521 
GS -0.0488775328831961 -0.0135121943728409 ** 0.0377825479458521 
USB -0.107830473836386 0.0341711076492987 ** 0.0377825479458521 
CB -0.0901844998495054 -0.0985386278446791 ** 0.0377825479458521 
MS 0.00227834951905075 -0.0920449700517052 ** 0.0377825479458521 
AXP -0.104327766762045 -0.0162314701226256 ** 0.0377825479458521 
ICE -0.0811525508138042 -0.0378088620567959 ** 0.0377825479458521 
BBT -0.136991907923339 0.0280298699401075 ** 0.0377825479458521 
TRV -0.20168222412595 -0.0437715964556488 ** 0.0377825479458521 
SPGI -0.00645881348352848 -0.0335211126805885  0.0377825479458521 
AON -0.113506441431514 -0.0262821934708995 ** 0.0377825479458521 
AFL -0.203097367224819 0.10711590165834 ** 0.0377825479458521 
ALL -0.109974287818419 -0.0204586085995221 ** 0.0377825479458521 
STT -0.146763172622148 -0.0299751300500908 ** 0.0377825479458521 
STI -0.0612653830209914 0.0213140855465753 ** 0.0377825479458521 
DFS -0.0836161625072747 -0.00547529490921607 ** 0.0377825479458521 
RF -0.00810885842382611 -0.073715591613829 ** 0.0377825479458521 
PFG -0.0477039562119986 -0.0497092833824634 ** 0.0377825479458521 
TROW -0.123655292200166 -0.0457198993648533 ** 0.0377825479458521 
WLTW -0.0372185135152648 -0.0824654326904595 ** 0.0377825479458521 
LNC -0.0462211872989022 -0.00166199148230785 ** 0.0377825479458521 
BEN -0.0523441200109036 -0.0424716019423321 ** 0.0377825479458521 
HBAN -0.082539241511326 0.0504037793277749 ** 0.0377825479458521 
L -0.15931989931682 0.00298517418800225 ** 0.0377825479458521 
IVZ -0.113467825133864 -0.00714649451463555 ** 0.0377825479458521 
CMA -0.0724396726661677 -0.00314028752852232 ** 0.0377825479458521 
CINF -0.174034521842851 -0.0751836604271167 ** 0.0377825479458521 
UNM -0.156895756787395 -0.0334000841127673 ** 0.0377825479458521 
XL -0.0171271951898044 -0.0163243178916789  0.0377825479458521 
AJG -0.0910118583535343 0.00217679533424188 ** 0.0377825479458521 
ETFC -0.00508780255018044 -0.00809512138027311  0.0377825479458521 
RJF -0.0797183553944291 -0.0157614629957661 ** 0.0377825479458521 
AMG -0.0563830729562557 0.0341842130634544 ** 0.0377825479458521 
LUK -0.047614023063904 0.00217716302892084 ** 0.0377825479458521 
TMK -0.111065929777258 0.0326710381984407 ** 0.0377825479458521 
ZION -0.0281758159340018 0.00889686303588805  0.0377825479458521 
NDAQ -0.0338723081842863 0.00430166275574087  0.0377825479458521 
CBOE -0.0532537993511472 0.066195481651599 ** 0.0377825479458521 
PBCT -0.130790709218224 -0.0454650521456619 ** 0.0377825479458521 
AIZ -0.0949937122321678 0.000423970731246187 ** 0.0377825479458521 
NAVI 0.0547632105906803 0.0330805652408505 ** 0.0377825479458521 
DOW -0.0447964173307365 0.0326236330615458 ** 0.0377825479458521 
DD -0.0270185239565034 -0.00945384927156765  0.0377825479458521 
MON -0.0217856893889407 -0.0459447232549948 ** 0.0377825479458521 
PX -0.066050143297816 -0.0471271970062459 ** 0.0377825479458521 
ECL -0.123055887638274 -0.0535958188047616 ** 0.0377825479458521 
PPG -0.0633282488831463 -0.0040491947167132 ** 0.0377825479458521 
APD -0.0316480402673555 -0.0405969407828967 ** 0.0377825479458521 
SHW -0.0785716358774839 -0.0512791442029955 ** 0.0377825479458521 
LYB 0.0561451545093441 -0.000588850046618275 ** 0.0377825479458521 
IP -0.0105610870308756 0.00995591710051536  0.0377825479458521 
NUE -0.0873056229943715 -0.0475370688748524 ** 0.0377825479458521 
NEM -0.0507198195580616 -0.041630987947113 ** 0.0377825479458521 
FCX 0.00486664807471653 0.000560967869591787  0.0377825479458521 
VMC 0.0437786853909507 -0.0344001785221484 ** 0.0377825479458521 
MLM 0.0549278795827485 -0.0488336695194519 ** 0.0377825479458521 
WRK 0.0629294383727902 -0.0345431185655866 ** 0.0377825479458521 
BLL -0.0334548891170788 0.000796125617868546  0.0377825479458521 
ALB -0.0264145512298628 -0.00625238608542115  0.0377825479458521 
EMN -0.0202267607402485 0.0322732327992862  0.0377825479458521 
IFF -0.119842547097384 -0.00827610696392747 ** 0.0377825479458521 
FMC -0.0370767959951041 -0.0593281817912048 ** 0.0377825479458521 
MOS 0.00406965753756472 -0.0522120992517892 ** 0.0377825479458521 
SEE -0.057037025303748 0.00294223646290396 ** 0.0377825479458521 
AVY -0.0382560831005411 -0.014865793751329 ** 0.0377825479458521 
CF -0.00819915657977258 -0.0267796065409429  0.0377825479458521 
GE -0.023244578854324 0.0265384987529405  0.0377825479458521 
MMM -0.0702718177889118 -0.0387309779906187 ** 0.0377825479458521 
BA 0.0109590179775202 -0.0287398438093762  0.0377825479458521 
HON -0.0383693902394317 -0.0272233879393427 ** 0.0377825479458521 
UNP -0.0244316450475462 -0.0381671014056369 ** 0.0377825479458521 
UTX -0.0693525798275395 -0.043820747096321 ** 0.0377825479458521 
UPS -0.0279300747189551 -0.045113354050877 ** 0.0377825479458521 
LMT -0.10038530660541 0.0158725195527435 ** 0.0377825479458521 
CAT -0.000321459155340597 0.0116655936661756  0.0377825479458521 
GD -0.066642353961703 0.0298917423438843 ** 0.0377825479458521 
LUV -0.0901347145252602 0.0454966469916919 ** 0.0377825479458521 
DE -0.00694818710365999 -0.0313505790363351  0.0377825479458521 
DAL 0.0144097754301961 -0.00334060868464342  0.0377825479458521 
WM -0.0469326014292097 -0.074509016455177 ** 0.0377825479458521 
PCAR -0.0412887481161378 -0.0261495888811545 ** 0.0377825479458521 
PH -0.0125301405818139 -0.011895799768956  0.0377825479458521 
ROK -0.0184318353080831 -0.0335231906937113  0.0377825479458521 
EFX -0.0550905181828341 -0.0251152608541793 ** 0.0377825479458521 
AAL 0.0344720678576242 0.0229593629250087  0.0377825479458521 
IR -0.0310847587221513 -0.0338572346488253  0.0377825479458521 
ROP -0.0675634268257082 -0.077704067828192 ** 0.0377825479458521 
FTV 0.0488513543420545 -0.0159982194050378 ** 0.0377825479458521 
SWK -0.0162788373629927 -0.0372662185725849  0.0377825479458521 
UAL 0.0690880093261351 -0.019119551896606 ** 0.0377825479458521 
COL -0.0527101601588631 -0.00192083825188458 ** 0.0377825479458521 
GWW -0.00970797857749463 -0.0122562361543326  0.0377825479458521 
TXT 0.00209496116030809 0.0155915114936003  0.0377825479458521 
DOV -0.0326251614290167 0.0251972147288209  0.0377825479458521 
FAST -0.00372987804438142 -0.0130190550319862  0.0377825479458521 
RSG -0.0266069263577563 -0.00796932299774502  0.0377825479458521 
FLR -0.0448232530998872 -0.0484635302650482 ** 0.0377825479458521 
PNR 0.0123914700448528 -4.82883083239125e-05  0.0377825479458521 
URI 0.0616866203993681 0.0245873040471235 ** 0.0377825479458521 
EXPD -0.0926928002883364 0.00707231040187657 ** 0.0377825479458521 
SNA -0.0110889016354262 -0.00988163103407967  0.0377825479458521 
KSU -0.0117639584092966 -0.00158763232308502  0.0377825479458521 
FBHS 0.0289240438447746 -0.00420856745949863  0.0377825479458521 
XYL 0.00642142718597881 0.0191288494989401  0.0377825479458521 
RHI -0.053933300547841 -0.0163049536314839 ** 0.0377825479458521 
JBHT -0.0860705353443015 0.0134936519911706 ** 0.0377825479458521 
AYI -0.0262108069085176 0.000100646928523548  0.0377825479458521 
R -0.033569790585108 0.0260183957056704  0.0377825479458521 
ALLE 0.071791978139639 0.00209361954650035 ** 0.0377825479458521 
SRCL -0.0454391900416325 -0.0214081211354089 ** 0.0377825479458521 
JEC -0.032378232182086 -0.0268039320627433  0.0377825479458521 
FLS -0.0223722352739972 -0.0409351498246482 ** 0.0377825479458521 
PWR -0.0271993063028937 -0.029467008401664  0.0377825479458521 
HES -0.0382046321294291 -0.0570669865098722 ** 0.0377825479458521 
CVX -0.0997064687533108 -0.0573128819839226 ** 0.0377825479458521 
COP -0.0456617159454092 -0.0586681694691485 ** 0.0377825479458521 
OXY -0.0817297336229358 -0.0674534813906045 ** 0.0377825479458521 
XOM -0.154390931073284 -0.0937170896373303 ** 0.0377825479458521 
TOT -0.0410477829041835 -0.0726442007287631 ** 0.0377825479458521 
APC -0.0191200093891514 -0.0486279651769037 ** 0.0377825479458521 
RDS-A -0.0492678418517147 -0.0694310181399589 ** 0.0377825479458521 
BP -0.0373603425230508 -0.063366917972029 ** 0.0377825479458521 
MRO -0.0290751816452744 -0.0529267122907142 ** 0.0377825479458521 
MPC 0.017306189764944 0.0449958365717394 ** 0.0377825479458521 
VLO -0.00345941128970282 -0.0602516199610069 ** 0.0377825479458521 
SUN 0.0852300471426968 -0.00491511937371354 ** 0.0377825479458521 
EC -0.0869753722946993 0.0287433678424457 ** 0.0377825479458521 
NBL -0.0502316224336053 -0.0471790215190326 ** 0.0377825479458521 
PBR 0.00342045429725232 -0.0508831766577323 ** 0.0377825479458521 
PSX 0.0692399362629805 0.0115581213823195 ** 0.0377825479458521 
PTR -0.0582862510082401 -0.0126595761879041 ** 0.0377825479458521 
STO -0.0731744314134723 -0.0378427683283295 ** 0.0377825479458521 
SU 0.00296420149908658 -0.0204177001582495  0.0377825479458521 
XOI -0.0825226884489279 -0.0714079178125618 ** 0.0377825479458521 
USO -0.0504181271525021 0.00401645415989441 ** 0.0377825479458521 
UWTI -0.011906902707701 0.0027545420837071  0.0377825479458521 
UCO -0.00928451932284817 0.0112879344467586  0.0377825479458521 
XLE -0.0882425761390582 -0.0674070863899576 ** 0.0377825479458521 
OIL -0.0355705914571595 0.0190243100411833  0.0377825479458521 
DWTI -0.0554859100583393 0.023015554550779 ** 0.0377825479458521 
SCO 0.00853833435477296 -0.0258144935527226  0.0377825479458521 
DBO -0.0512136079127417 0.00906204028306875 ** 0.0377825479458521 
VDE -0.0876327799129898 -0.0546057355975147 ** 0.0377825479458521 
DIG -0.0880309307891684 -0.0603871584630796 ** 0.0377825479458521 
NEE -0.0733644460612461 -0.0636552503166962 ** 0.0377825479458521 
DUK -0.00997790236373513 -0.00529897668985917  0.0377825479458521 
SO -0.101100910020167 -0.0692341551544639 ** 0.0377825479458521 
D -0.049134974763487 -0.0348094820916262 ** 0.0377825479458521 
PCG -0.150077912840686 -0.0219647497342976 ** 0.0377825479458521 
EXC -0.077282663522739 -0.0274691854350439 ** 0.0377825479458521 
AEP -0.0980825075902536 -0.0518454974809491 ** 0.0377825479458521 
SRE -0.104596811864245 -0.0279970190959245 ** 0.0377825479458521 
EIX -0.11419085298171 -0.0222443605371055 ** 0.0377825479458521 
PPL -0.0887592909810332 -0.0523912137241822 ** 0.0377825479458521 
ED -0.0887189918161469 -0.0335470296467105 ** 0.0377825479458521 
PEG -0.0747726744199105 -0.0481296763548322 ** 0.0377825479458521 
XEL -0.128560310954468 -0.0666299802112818 ** 0.0377825479458521 
WEC -0.0602435785132331 -0.026441505233539 ** 0.0377825479458521 
ES -0.0874314508122689 0.00353755086428073 ** 0.0377825479458521 
DTE -0.104562167411666 -0.00966464871970302 ** 0.0377825479458521 
AWK -0.102357501708293 -0.0285113646928926 ** 0.0377825479458521 
FE -0.0898835455866787 -0.043179874198026 ** 0.0377825479458521 
ETR -0.0652079121330536 -0.059541223895436 ** 0.0377825479458521 
AEE -0.0653698480558247 -0.0338633264007653 ** 0.0377825479458521 
CMS -0.0499171452392984 -0.016127889320154 ** 0.0377825479458521 
CNP -0.0825241291207523 -0.00834062045595049 ** 0.0377825479458521 
PNW -0.0425099176115307 -0.050167351202929 ** 0.0377825479458521 
SCG -0.0911646488712884 -0.0270481278238912 ** 0.0377825479458521 
LNT -0.0612268912250421 -0.00413924863994639 ** 0.0377825479458521 
NI -0.0434609243051964 0.00455012747456402 ** 0.0377825479458521 
AES -0.105010621208799 -0.0170512227879755 ** 0.0377825479458521 
NRG -0.00531926383708485 -0.0442995171959506 ** 0.0377825479458521 
AMZN -0.0173431941000491 -0.0572909606926886 ** 0.0377825479458521 
HD 0.0102997620394902 -0.0391140195542885 ** 0.0377825479458521 
CMCSA -0.0431592089011311 -0.0305526266988228 ** 0.0377825479458521 
DIS -0.0654052775183216 -0.058509299824095 ** 0.0377825479458521 
MCD -0.0555431302523456 -0.0725231924649655 ** 0.0377825479458521 
PCLN -0.0634494242545721 0.0107554334704399 ** 0.0377825479458521 
SBUX -0.0264840118856933 0.0090282981493281  0.0377825479458521 
TWX -0.0276886333570597 0.0475769380678315 ** 0.0377825479458521 
NKE -0.0216340279756836 -0.0173834102817006  0.0377825479458521 
LOW 0.00066872888024115 -0.0369036794774796  0.0377825479458521 
CTH 0.00884617050635157 0.031890535278314  0.0377825479458521 
NFLX -0.00676645960178028 -0.0160997946559936  0.0377825479458521 
TJX -0.0549221138152049 -0.0440488347697457 ** 0.0377825479458521 
GM 0.0126807900556471 0.0102000729827808  0.0377825479458521 
F 0.0415812775557277 0.0995358474777039 ** 0.0377825479458521 
FOXA -0.0822423528339217 -0.0212921981857 ** 0.0377825479458521 
TGT -0.0517320727470442 -0.0825632334214347 ** 0.0377825479458521 
MAR -0.0290288421373084 -0.00261505944540928  0.0377825479458521 
CBS -0.0235727919393049 -0.000344489102640036  0.0377825479458521 
ROST -0.0286880757686175 -0.0491646056918604 ** 0.0377825479458521 
CCL -0.0246254638062003 -0.061594976397381 ** 0.0377825479458521 
ORLY -0.0528844464491483 -0.0293271090910932 ** 0.0377825479458521 
NWL -0.0155215194456777 0.0152919544924001  0.0377825479458521 
YUM -0.0292497257303831 -0.0532432598325073 ** 0.0377825479458521 
DLPH 0.0416498008901308 -0.0200021444983505 ** 0.0377825479458521 
AZO -0.0886370569339614 0.00766062914437811 ** 0.0377825479458521 
OMC -0.0531651420852231 -0.0284637048325693 ** 0.0377825479458521 
DLTR -0.0195435399576827 -0.0382973610946264 ** 0.0377825479458521 
VEC –0.115501834436348 -0.00635348221904064 ** 0.0377825479458521 
DG -0.0568487334584326 -0.0618312533421364 ** 0.0377825479458521 
ULTA 0.0215435274809328 -0.0334928968433729  0.0377825479458521 
RCL 0.0172882860622098 -0.0134788300275738  0.0377825479458521 
EXPE 0.0175299892936802 -0.0294168472279586  0.0377825479458521 
VIAB -0.0308721226570784 -0.0505405774132897 ** 0.0377825479458521 
FOX -0.0612565690278856 -0.0218127117895231 ** 0.0377825479458521 
MHK 0.0304864250649591 0.0289257732592913  0.0377825479458521 
DISH -0.0266557631345965 -0.0606702801279222 ** 0.0377825479458521 
GPC -0.0128440729958345 -0.0357158342236289  0.0377825479458521 
BBY 0.00669609216988894 -0.0185067092040432  0.0377825479458521 
CMG 0.0176195720624397 -0.0243915478168143  0.0377825479458521 
WHR -0.00874080415080197 0.0221609435960056  0.0377825479458521 
DHI 0.0263051746607446 -0.0068813101076015  0.0377825479458521 
COH 0.0265720056186497 -0.0258865516582523  0.0377825479458521 
HOG 0.0156924583515673 0.00190001304551743  0.0377825479458521 
HAS -0.0543252553770332 -0.0221822886574684 ** 0.0377825479458521 
KMX 0.0189937519222326 -0.012172432740949  0.0377825479458521 
AAP -0.0418942194583518 -0.011105434688528 ** 0.0377825479458521 
LEN 0.049235300267932 0.0104785627040376 ** 0.0377825479458521 
DRI 0.0206198599928319 -0.0229798234032623  0.0377825479458521 
TIF 0.00944257329284317 -0.051428493434914 ** 0.0377825479458521 
IPG -0.0456147146511072 -0.00332445095408379 ** 0.0377825479458521 
FL 0.0110019480885131 0.00438578917933882  0.0377825479458521 
TSCO -0.0252462699503863 -0.024027167660071  0.0377825479458521 
WYNN 0.033927148499723 0.0123487581629551  0.0377825479458521 
GT 0.0062491209153798 0.00658751578275136  0.0377825479458521 
M -0.003162320345821 -0.0147536737453199  0.0377825479458521 
WYN 0.0409865627750305 -0.0125477937449198 ** 0.0377825479458521 
MAT -0.00738356388489567 -0.00805128722025842  0.0377825479458521 
LKQ -0.00873231364779504 -0.00633991016303413  0.0377825479458521 
BBBY -0.0526880853264407 -0.0292907852942034 ** 0.0377825479458521 
GRMN -0.00747527469310973 -0.0378346016177086 ** 0.0377825479458521 
TGNA 0.0185173388050813 0.0968054785086797 ** 0.0377825479458521 
JWN 0.0332175510971828 -0.00296979041439022  0.0377825479458521 
HRB -0.035687132031631 -0.0362902424037047  0.0377825479458521 
NWSA 0.000859780724714517 -0.00568826543446071  0.0377825479458521 
TRIP 0.0243880717676733 -0.0165470942842485  0.0377825479458521 
SIG 0.00918683385779967 -0.0328087849872561  0.0377825479458521 
DISCA -0.0560026907004845 -0.028227402184775 ** 0.0377825479458521 
RL 0.0228440097264209 -0.0267600822646427  0.0377825479458521 
UAA 0.0402818535111931 -0.0253546227288141 ** 0.0377825479458521 
UA 0.0925081925245793 0.0502649508097967 ** 0.0377825479458521 
AN 0.0400102875897468 0.00306388828179625 ** 0.0377825479458521 
NWS 0.01151118947946 -0.00112980586012625  0.0377825479458521 
AAPL -0.00344207668544184 -0.0148069328186942  0.0377825479458521 
MSFT -0.0581537199382001 -0.0514672629520163 ** 0.0377825479458521 
FB 0.0242040601272429 0.027649514061513  0.0377825479458521 
GOOGL 0.00499256723689444 -0.00288336054303166  0.0377825479458521 
T -0.0558258724297197 -0.106244747388746 ** 0.0377825479458521 
INTC -0.0769089369469858 -0.0306084665712628 ** 0.0377825479458521 
V -0.0695821639962481 -0.0305208236779097 ** 0.0377825479458521 
CSCO -0.041600489962587 -0.0444930656310948 ** 0.0377825479458521 
VZ -0.0188935435789886 -0.0650735813986673 ** 0.0377825479458521 
PYPL 0.09196383835724 0.0232180108755843 ** 0.0377825479458521 
ADP -0.129667648977007 -0.0282598370736624 ** 0.0377825479458521 
AMAT -0.0539405646518201 0.00536355637279664 ** 0.0377825479458521 
YHOO -0.0422364373172396 -0.0331947069113769 ** 0.0377825479458521 
EBAY -0.0654061850385771 -0.0251399718866078 ** 0.0377825479458521 
ATVI -0.0630574483780942 -0.0586044370463385 ** 0.0377825479458521 
HPE -0.000804264442916011 -0.0231348550563911  0.0377825479458521 
HPQ -0.0360843351027379 0.0147814912109553  0.0377825479458521 
ADI -0.0655377705885021 -0.0358863802046437 ** 0.0377825479458521 
INTU -0.108884882393872 -0.0263511428856961 ** 0.0377825479458521 
MU 0.0318149385111538 -0.0160916287943178  0.0377825479458521 
EA -0.0336118016826927 -0.00514166746900812  0.0377825479458521 
FIS -0.0812943583264354 -0.0112350596925837 ** 0.0377825479458521 
TEL 0.00724522081214939 -0.00775073214442354  0.0377825479458521 
WDC -0.0515400539007221 0.0280207141144566 ** 0.0377825479458521 
FISV -0.074437487088425 0.0109475265441429 ** 0.0377825479458521 
GLW 0.00763746872466257 -0.0129224858793486  0.0377825479458521 
APH -0.0392022437248607 -0.036309484092678 ** 0.0377825479458521 
LRCX -0.0334379596094094 0.00255912866730621  0.0377825479458521 
DXC -0.0293386065707787 -0.0145935713427751  0.0377825479458521 
SYMC -0.0797656290360084 -0.0529522028888647 ** 0.0377825479458521 
PAYX -0.0646406762666813 -0.026688358675993 ** 0.0377825479458521 
SWKS -0.0311550181384956 0.00181505128643426  0.0377825479458521 
ADSK -0.029390239780121 0.0021139564698393  0.0377825479458521 
LVLT -0.00353992109011977 0.00457491426518462  0.0377825479458521 
MCHP -0.0378250201132974 0.00321766232223941 ** 0.0377825479458521 
KLAC -0.031161212161505 0.0196804003024459  0.0377825479458521 
RHT -0.0152496632208506 -0.0216184711568318  0.0377825479458521 
STX 0.0153258023156781 0.0334379162904983  0.0377825479458521
Attachments area

M

Read Full Post »

My sister committed suicide yesterday.  I am so shocked that even my grief cannot express itself.  Unlike when my father died in 2005 and my reaction was an anguished cry this event leaves me drained of all feeling with periodic waves of a feeling of infinite loss so destabilizing for my psyche that I am too fearful to feel the loss.  Unfortunately, I can no longer believe with any conviction that there is eternal life somewhere.  She was a beautiful tortured soul.  I wish I did not have a trainwreck life and thus somehow be in a more financially secure and stable situation since I believe that I could have done something to avoid this situation but nothing brings home my powerlessness in this situation.  Grief so tremendous hides within that I cannot face it.

Read Full Post »

zulfikar.ahmed@gmail.com <zulfikar.ahmed@gmail.com>

Attachments8:48 PM (10 hours ago)

to harrington, jharris, jhp, jhricko_4, jianjunp, jinha, jjbrehm, jlind, jlondon, jlw, jmateo, jmerseth, jmetcalf, jmg, jmogel, joel, joelms, john.aldrich, john.beatty, johncrawford53, jose.oliveira, josesoto, josue, jpadgett, jpbalz
Ladies and Gentlemen,
We are in the PRESCIENTIFIC era of finance.  The preeminent financial services companies of the world, such as Goldman Sachs, doesn’t know finance.  What do I mean by this?  Being lazy, I refer you to the book by George Soros, ‘The Alchemy of Finance’ because why repeat good work already done.  He will explain to you why financial markets are not like natural phenomena studied in physics because thinking participants affect the system being studied.  So Goldman Sachs does not know finance.  Neither do economists at Yale and Harvard.  The academics are still confused from 1970s championing random walks (which was postulated by Bachelier in 1900 in the more sophisticated continuous time Brownian motion).  Then the 1990s they are slowly talking about behavioral finance which unfortunately is in a messy state where no one has clear quantitative models that can be used.  

Now Soros is a genius of first order but his theory of reflexivity is quite elaborate and not quite quantitative in usable forms.  I have been thinking about finance since 1995 from my first job at Lehman and it is becoming clearer to me that this is actually completely open terrain.  Consider volatility.  No one really knows what volatility is.  I mean it’s easy to compute standard deviation of returns and define this as volatility but there is a way in which the WORD means something more and something deeper about, well the measure of volatile-ity.  Well, if you want to measure volatile-ity, standard deviation of returns is a pretty stupid measure.  So Mandelbrot brought into the fray the beautiful empirical observation of long memory in the markets.  This is interesting because long memory originally was discovered in hydrology, with measurements of water levels of Nile.  To make a long story short, standard deviation volatility (say proxied by return^2) has long memory.  There are many studies about this.  I FINALLY realized today that long memory of volatility is the totally wrong thing to consider.  The right thing to consider is the long memory of the VOLUME OF TRADE.  In particular if you take say the percent of total shares being traded (volume/shares outstanding) for stocks, you can see the long memory of this quantity directly say by calculating the autocorrelation sequence at different lags.  THIS is the right object for long memory.  In other words, the total volume traded has long memory, and this is a direct measurement while it SO HAPPENS that volume and the standard deviation of returns volatility HAPPEN to be correlated and so it is a CONSEQUENCE that there is long memory of volatility.  The volume traded has long memory and this is the deeper issue because whatever we mean by volatility is happening under the volume or behind it.  Yes yes Goldman folks know this but they don’t know finance anyway because there are no clear models that include reflexivity phenomena that figure as doctrine in any investment bank.  Besides, they cannot break out of the religions that they earn their living on, which are completely wrong things.

The first thing we need to understand from Soros is that various disequilibria that are the fundamental features of financial markets find their most basic examples in the volume series concretely.  Soros’ concepts are based not on equilibrium prices (which he explains are meaningless) and on a reflexivity of two tendencies (he borrows heavily from Hegelian dialectics of thesis/antithesis) he calls the cognitive function and the participative function of market participants.  The first place to look for this is of course VOLUME TRADED.  So we immediately find that the long memory effects are all to be found here as well.  So to the extent that Goldman Sachs does not explicitly model the markets with these sorts of features (they can’t because they sell index funds etc.) they don’t know finance.

First I will share with you my latest great discovery, which I can say without much fake humility is absolute genius:  even the most ELEMENTARY use of quantitative measures of reflexivity ideas leads very quickly to the discovery that almost EVERY oil stock (and probably much of the equities markets) has NEGATIVE autocorrelation at 1 day lag which while not super strong is strong enough that we can generate extremely good mean reversion strategies for them without doing pairs.  So while I had a slightly more involved Alpha for Every Sector strategies before, here is a concrete and deeper example of how to produce alpha for every stock in the oil sector (XLE components) using mean reversion and trend following.  The period is 2007-2017 and it is quite good in almost every single stock in the oil sector.  So that’s the first great discovery, that alpha is quite common although it takes a bit of care to get the quantitative strategy to be in good order (see the R code for details; you will need the code in the attached file to get this to work in your machine):

meanRev4<-function(rets,volchanges){ 
  n<-length(rets);
  vRef<-rep(0,n)
  v<-rep(0,n)
  
  updown<-1
  lastChange<-10000
  for(k in 302:n){ 
    lastChange<-lastChange+1
    q<-quantile(volchanges,c(0.5,0.5))
    
    CVRef<-cumsum(vRef)
    CV<-cumsum(v)
    CVRefSmooth<-savgol(CVRef,301,forder=3,dorder=0)
    avgVeryLong<-mean(CVRef[(k-30):(k-1)])
    avgLong<-mean(CVRef[(k-5):(k-1)])
    avgShort<-CVRef[k-1]
    
    actualAvgLong<-mean(CV[(k-5):(k-1)])
    actualAvgShort<-CV[k-1]
    #if (avgLong>1.005*avgShort && lastChange>0){
    #  updown<-  -1
    #  lastChange<-0
    #}
    #if (avgLong<0.995*avgShort && lastChange>0){
    #  updown<-  1
    #  lastChange<-0
    #}
    
    
    updown<- -sign(CVRefSmooth[k-1]-CVRefSmooth[k-2])
    #if (abs(avgLong-avgShort)<0.0005){
    #  updown<- 0
    #  lastChange<-0
    #}
    
    if( volchanges[k-1]>q[2] || volchanges[k-1]<q[1] ){
      vRef[k]<- -sign(rets[k-1])*rets[k]
      v[k]<-updown*vRef[k]
      if (v[k]< -0.02) {
        v[k]<- -0.02
      }
 
            
    }
  } 
  list(v=v,vRef=vRef)
}
performanceTrendFollowingMeanReversion<-function( symbols,startDate ) {
  R<-constructComponentRets(symbols,startDate)
  V<-constructVolumeChanges(symbols,startDate)
  R[is.na(R)]<-0
  V[is.na(V)]<-0
  dts<-index(R)
  multiseries<-NULL
  nsymb<-length(symbols)
  for (ns in 1:nsymb){
    x<-meanRev4(coredata(R[,ns]),coredata(V[,ns]))
    multiseries<-cbind(multiseries,x$v)
  }
  df<-data.frame(multiseries,index=dts)
  names(df)<-symbols
  matplot( df,type=c(‘l’))
  legend(‘topleft’,legend=1:nsymb,col=1:nsymb,pch=1)
  df
}
Now the second consequence of the problem of Goldman Sachs (and therefore the rest of the industry) not knowing finance.  Just as Bill Gates understood that no one knew software in 1970s we must understand that despite a century of theory of finance since Bachelier and some fancy models, no one actually knows finance because the two fundamental directions (fundamental/technicals) are inadequate to have any real clue about what is going on in the actual reality.  The answer is that someone (if not me someone at least) should start a TECHNOLOGY STARTUP that does this right: the only way that any serious WIDESPREAD OPEN understanding/use of finance can take shape is if we can COMMODITIZE QUANTITATIVE MODELS directly in retail so that actual models are easy to extract and EXECUTE in the markets.  Finance cannot be understood without quantitative models being accessible for RETAIL USE WORLDWIDE and the right vehicle for this is a technology company that provides this sort of service.  The correct quantitative models of the markets should not be considered material for the cognoscenti.  It should be provided as a retail service more or less how Google and Yahoo provide stock data but with ease of EXECUTION of strategies that are solid by anyone anywhere.  This is a big idea in the sense that such a company will probably make the entire financial services investment banks obsolete in time.  Just as Bill Gates understood that it was SOFTWARE that was more important than hardware, we must understand that in the situation where no one knows any finance, the first company to provide PUBLIC OPEN ACCURATE MODELS that show clear alpha and which can be executed conveniently by anyone anywhere in the world will be the company that redefines all financial markets.  The idea is that in the case where no one knows finance (and I don’t want to waste time arguing this since I can cite Soros’ book) the first technology company that makes concrete open easily accessible STANDARD for models of markets which can be USED easily and UNDERSTOOD easily and which which are OPEN and robust etc. will define for posterity what finance is.  Alpha in the markets are ubiquitous but it is quite a bit of work to build the models for the first time by hand.
The individual stock strategies are quite good alpha so check them out by running the code.  The Alchemy of Finance must be accessible to seven billion of this planet in a concrete manner.  A technology company is the right way to do this.  It will succeed if done properly more than Google and Facebook.
For sector strategies, I got decent results by machine learning.  Some of those graphs are attached.  This is original work so unpolished.  QUANTITATION of reflexivity ideas may seem at first glance like technicals/chartist strategies but this is not the right way to think about this.  The chartists’ concepts and viewpoint are not the right way to consider the issues of what is happening in the markets.  Soros’ ideas are much more serious but at the same time, we need some concrete simple results that are clear and compelling.  Alpha in the stock markets should be UNDERSTOOD, should be GLOBALLY PUBLIC, should be available to everyone in the world to trade.  Until this happens, this is a great way to establish a great new company; later on we can understand whether a true science of finance has a chance of emerging.  Soros is saying that this will be forever impossible in a sense but on the other hand, we can’t be sure until we try.

Financial markets should be available to everyone with convenient ways of trading without having to get a Masters in Finance.  This is a great opportunity to transform the entire structure of global finance as well as a path to addressing some subtler issues such as VOLATILITY STORMS and other deeper instabilities.  Academic ways of dealing with these is not going to be effective.  We don’t know what volatility is; how will we be able to quell volatility storms (which I coined but you can probably make sense of these from Soros’ theory).

12 Attachments

Read Full Post »

Even with a spread that has an autoregressive AR(1), prediction much better than 50% is hard with standard machine learning strategies.  For technology sector XLK, I created minimal lag 1 autocorrelation portfolios per day then I use variables 6 days of lagged returns of the chosen spread, the correlation, the long memory parameter for volatility, the acf.  The prediction accuracy by machine learning algorithm is:

Nearest Neighbors 0.501735250372
Linear SVM 0.5096678235
RBF SVM 0.506197322757
Gaussian Process 0.506693108577
Decision Tree 0.491819533961
Random Forest 0.513634110064
Neural Net 0.513138324244
AdaBoost 0.486365889936
Naive Bayes 0.489340604859
QDA 0.523549826475

So in this group quadratic discriminant analysis does best but none of these predictions are particularly spectacular.  The code:

import pandas.io.data as web
import pandas as pd
import numpy as np
import sys
import pykalman
from datetime import datetime,timedelta
from math import isnan
import matplotlib.pyplot as plt
import statsmodels.api as sm
from sklearn import linear_model
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.gaussian_process import GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
import warnings
warnings.filterwarnings("ignore")

Sectornames=["XLY",
 "XLP",
 "XLE",
 "XLF",
 "XLV",
 "XLI",
 "XLB",
 "XLRE",
 "XLK",
 "XLU"]
sectors=["Consumer Discretionary",
 "Consumer Staples",
 "Energy",
 "Financials",
 "Health Care",
 "Industrial",
 "Materials",
 "Real Estate",
 "Technology",
 "Utilities"]


XLBSymbols=["DOW",
 "DD",
 "MON",
 "PX",
 "ECL",
 "PPG",
 "APD",
 "SHW",
 "LYB",
 "IP",
 "NUE",
 "NEM",
 "FCX",
 "VMC",
 "MLM",
 "WRK",
 "BLL",
 "ALB",
 "EMN",
 "IFF",
 "FMC",
 "MOS",
 "SEE",
 "AVY",
 "CF"]

XLUSymbols=["NEE",
 "DUK",
 "SO",
 "D",
 "PCG",
 "EXC",
 "AEP",
 "SRE",
 "EIX",
 "PPL",
 "ED",
 "PEG",
 "XEL",
 "WEC",
 "ES",
 "DTE",
 "AWK",
 "FE",
 "ETR",
 "AEE",
 "CMS",
 "CNP",
 "PNW",
 "SCG",
 "LNT",
 "NI",
 "AES",
 "NRG"]

XLKSymbols=["AAPL",
 "MSFT",
 "FB",
 "GOOGL",
 "T",
 "INTC",
 "V",
 "CSCO",
 "VZ",
 "PYPL",
 "ADP",
 "AMAT",
 "YHOO",
 "EBAY",
 "ATVI",
 "HPE",
 "HPQ",
 "ADI",
 "INTU",
 "MU",
 "EA",
 "FIS",
 "TEL",
 "WDC",
 "FISV",
 "GLW",
 "APH",
 "LRCX",
 "DXC",
 "SYMC",
 "PAYX",
 "SWKS",
 "ADSK",
 "LVLT",
 "MCHP",
 "KLAC",
 "RHT",
 "STX"]


XLRESymbols=["SPG",
 "AMT",
 "CCI",
 "PSA",
 "EQIX",
 "PLD",
 "HCN",
 "WY",
 "AVB",
 "EQR",
 "VTR",
 "BXP",
 "VNO",
 "DLR",
 "O",
 "ESS",
 "HCP",
 "HST",
 "GGP",
 "MAA",
 "AIV"]

XLBymbols=["DOW",
 "DD",
 "MON",
 "PX",
 "ECL",
 "PPG",
 "APD",
 "SHW",
 "LYB",
 "IP",
 "NUE",
 "NEM",
 "FCX",
 "VMC",
 "MLM",
 "WRK",
 "BLL",
 "ALB",
 "EMN",
 "IFF",
 "FMC",
 "MOS",
 "SEE",
 "AVY",
 "CF"]

XLISymbols=["GE",
 "MMM",
 "BA",
 "HON",
 "UNP",
 "UTX",
 "UPS",
 "LMT",
 "CAT",
 "GD",
 "LUV",
 "DE",
 "DAL",
 "WM",
 "PCAR",
 "PH",
 "ROK",
 "EFX",
 "AAL",
 "IR",
 "ROP",
 "FTV",
 "SWK",
 "UAL",
 "COL",
 "GWW",
 "TXT",
 "DOV",
 "FAST",
 "RSG",
 "FLR",
 "PNR",
 "URI",
 "EXPD",
 "SNA",
 "KSU",
 "FBHS",
 "XYL",
 "RHI",
 "JBHT",
 "AYI",
 "R",
 "ALLE",
 "SRCL",
 "JEC",
 "FLS",
 "PWR"]


XLVSymbols=["JNJ",
 "PFE",
 "MRK",
 "UNH",
 "AMGN",
 "MDT",
 "ABBV",
 "CELG",
 "BMY",
 "GILD",
 "AGN",
 "LLY",
 "ABT",
 "TMO",
 "BIIB",
 "DHR",
 "AET",
 "ANTM",
 "ESRX",
 "SYK",
 "CAN",
 "ZTS",
 "ILMN",
 "HCA",
 "ZBH",
 "INCY",
 "EW",
 "BCR",
 "MYL",
 "CERN",
 "HOLX",
 "UHS",
 "DVA",
 "PRGO",
 "COO",
 "VAR",
 "EVHC",
 "PKI",
 "MNK",
 "PDCO"]

XLPSymbols=["PG",
 "PM",
 "KO",
 "MO",
 "WMT",
 "PEP",
 "CVS",
 "COST",
 "WBA",
 "CL",
 "TAP",
 "EL",
 "DPS",
 "MNST",
 "K",
 "CLX",
 "CAG",
 "MJN",
 "SJM",
 "HSY",
 "CPB",
 "MKC",
 "HRL",
 "CHD",
 "WFM",
 "BF-B",
 "COTY"]

XLFSymbols=["BRK-B",
 "JPM",
 "WFC",
 "BAC",
 "C",
 "GS",
 "USB",
 "CB",
 "MS",
 "AXP",
 "ICE",
 "BBT",
 "TRV",
 "SPGI",
 "AON",
 "AFL",
 "ALL",
 "STT",
 "STI",
 "DFS",
 "RF",
 "PFG",
 "TROW",
 "WLTW",
 "LNC",
 "BEN",
 "HBAN",
 "L",
 "IVZ",
 "CMA",
 "CINF",
 "UNM",
 "XL",
 "AJG",
 "ETFC",
 "RJF",
 "AMG",
 "LUK",
 "TMK",
 "ZION",
 "NDAQ",
 "CBOE",
 "PBCT",
 "AIZ",
 "NAVI"]

XLESymbols=["HES",
 "CVX",
 "COP",
 "OXY",
 "XOM",
 "TOT",
 "APC",
 "RDS-A",
 "BP",
 "MRO",
 "MPC",
 "VLO",
 "SUN",
 "EC",
 "NBL",
 "PBR",
 "PSX",
 "PTR",
 "STO",
 "SU",
 "XOI",
 "USO",
 "UWTI",
 "UCO",
 "XLE",
 "OIL",
 "DWTI",
 "SCO",
 "DBO",
 "VDE",
 "DIG"]

XLYSymbols=["AMZN",
 "HD",
 "CMCSA",
 "DIS",
 "MCD",
 "PCLN",
 "SBUX",
 "TWX",
 "NKE",
 "LOW",
 "CTH",
 "NFLX",
 "TJX",
 "GM",
 "F",
 "FOXA",
 "TGT",
 "MAR",
 "CBS",
 "ROST",
 "CCL",
 "ORLY",
 "NWL",
 "YUM",
 "DLPH",
 "AZO",
 "OMC",
 "DLTR",
 "VEC",
 "DG",
 "ULTA",
 "RCL",
 "EXPE",
 "VIAB",
 "FOX",
 "MHK",
 "DISH",
 "GPC",
 "BBY",
 "CMG",
 "WHR",
 "DHI",
 "COH",
 "HOG",
 "HAS",
 "KMX",
 "AAP",
 "LEN",
 "DRI",
 "TIF",
 "IPG",
 "FL",
 "TSCO",
 "WYNN",
 "GT",
 "M",
 "WYN",
 "MAT",
 "LKQ",
 "BBBY",
 "GRMN",
 "TGNA",
 "JWN",
 "HRB",
 "NWSA",
 "TRIP",
 "SIG",
 "DISCA",
 "RL",
 "UAA",
 "UA",
 "AN",
 "NWS"]



import re
sectorTicker=sys.argv[1]

def ret_ser(symbols,D,start,end):
 X=D[symbols].loc[start:end]
 R=np.diff(np.log(X))
 R[np.isnan(R)]=0
 return R

def print_full(x):
 pd.set_option('display.max_rows',len(x))
 print(x)
 pd.reset_option('display.max_rows')

def create_sector_strat_df( fName):
 print fName
 #regex=re.compile(r'.*(\d{4}-\d{2}-\d{2}) portA: ([^ ]+) portB: ([^ ]+) acf: ([^ ]+) d: ([^ ]+) cor: ([^ ]+) pr: ([^ ]*) dir: ([^ ]+) rc: ([^ ]+) rf: ([^ ]+) rp1: ([^ ]+) rp2: ([^ ]+) rp3: ([^ ]) rp4: ([^ ]+) rp5: ([^ ]+) rp6: ([^ ]+)\".*')
 regex = re.compile(r'.*(\d{4}-\d{2}-\d{2}) portA: ([^ ]+) portB: ([^ ]+) acf: ([^ ]+) d: ([^ ]+) cor: ([^ ]+) pr: ([^ ]*) dir: ([^ ]+) rc: ([^ ]+) rf: ([^ ]+) rp1: ([^ ]+) rp2: ([^ ]+) rp3: ([^ ]+) rp4: ([^ ]+) rp5: ([^ ]+) rp6: ([^ \"]+).*')
 colNames = ['date','acf','d','cor','pr','dir','rc','rf','rp1','rp2','rp3','rp4','rp5','rp6']
 df = pd.DataFrame( columns=colNames)
 df.set_index(['date'],inplace=True)
 with open(fName) as f:
 for line in f:
 m=regex.match(line)

 
 
 if m:
 date = datetime.strptime(m.group(1),'%Y-%m-%d')
 portA = m.group(2)
 portB = m.group(3)
 acf = m.group(4)
 d = m.group(5)
 cor = m.group(6)
 pr = m.group(7)
 dir = m.group(8)
 rc = m.group(9)
 rf = m.group(10)
 rp1 = m.group(11)
 rp2 = m.group(12)
 rp3 = m.group(13)
 rp4 = m.group(14)
 rp5 = m.group(15)
 rp6 = m.group(16)


 data = [[date,float(acf),float(d),float(cor),float(pr),float(dir),float(rc),
 float(rf),float(rp1),float(rp2),float(rp3),float(rp4),float(rp5),float(rp6)]]
 #print data
 ndf = pd.DataFrame( data, columns=colNames)
 df = df.append(ndf)
 return df


names = ["Nearest Neighbors", "Linear SVM", "RBF SVM", "Gaussian Process",
 "Decision Tree", "Random Forest", "Neural Net", "AdaBoost",
 "Naive Bayes", "QDA"]

classifiers = [
 KNeighborsClassifier(3),
 SVC(kernel="linear",C=0.0001),
 SVC(kernel="rbf",gamma=0.1),
 GaussianProcessClassifier(1.0*RBF(1.0),warm_start=True),
 DecisionTreeClassifier(max_depth=10),
 RandomForestClassifier(max_depth=10,n_estimators=10,max_features=1),
 MLPClassifier(alpha=10),
 AdaBoostClassifier(),
 GaussianNB(),
 QuadraticDiscriminantAnalysis()]

D = create_sector_strat_df( '%s.out' % sectorTicker)
D = D.dropna()
n=len(D.index)
Dp = D.ix[:,D.columns != 'date']
Dx = Dp.ix[:,Dp.columns != 'rf']
Dy = Dp['rf']

for name, clf in zip(names,classifiers):
 correct = 0
 total = 0
 lookback = 40
 for k in range( lookback,n-1):
 xtrain = Dx.iloc[(k-lookback):k].as_matrix()
 ytrain = np.sign(Dy.iloc[(k-lookback):k].as_matrix())

 model = clf
 model.fit(xtrain,ytrain)
 #print'score=', model.score(xtrain, ytrain)
 
 xtest = Dx.iloc[k+1].as_matrix()
 #print xtest
 yactual = Dy.iloc[k+1]
 
 predicted= model.predict(xtest)
 
 total = total + 1
 
 if np.sign(yactual) == np.sign(predicted[0]):
 correct = correct + 1

 print name, float(correct)/float(total)

 

Read Full Post »

Ladies and Gentlemen,

James Simons the mathematician co-responsible for the Chern-Simons invariants (integral of AdA + 2/3 A^3 for connection A on a compact three-manifold that Witten related to Jones polynomials of knots etc.)  suggests that the discoveries by the researchers at Renaissance Technologies of anomalies in markets are not worth publishing because they lack universality.  Of course universality in science is actually pretty rare as well.  Big Bang theory is hardly based on universal features and shares much more with the messy quantitative analysis of financial markets.  Anyway, let’s dig into an anomaly of a type that is extremely well-known.  You will learn how such a well-known type of anomaly does not disappear.  The efficient markets hypothesis is obviously a VERY rough approximation of the truth.  The laws of the markets may remain far more mysterious than the laws of the cosmos (whose mysteries in principle can be vastly lowered by removing the Big Bang cosmology and adopting Einstein static universe in principle).  The issue at hand is pairs trading on gold companies.  The first idea that occurs is that there is a gold mining index ETF and maybe trading components versus the index could be useful.  So here is the funny thing.  THIS idea does not do very well.  So for this idea, ARBITRAGE traders (at D E Shaw, Renaissance etc.) have cleaned up this part.  On the other hand a minor variation which I present here will be enormously profitable.  The variation is to use the index not for trading but to pick pairs.  Use 30-days to calculate CORRELATIONS between the components and the index.  Pick two: MAXIMUM correlated and MINIMUM correlated with the GDX index.  Then take position on the difference of the two stocks for a single day every day.  More precisely, run the following R code which gets the data, runs the strategy etc.  The Sharpe is 15.4 on S&P 500.  So this strategy is worth investing in by any respectable quant fund (who are reasonably happy to run Sharpe 4+).  But what this is telling us is that ARBITRAGEURS are UNABLE to close the gaps of this type which are ALREADY the focus of a great deal of attention.  In other words, EFFICIENT MARKET HYPOTHESIS is much like the HUMAN RATIONALITY HYPOTHESIS.  It’s extremely misleading.  Just as in the Enlightenment there was a hope of human rationality becoming transcendent in some way which then led to not much since rationality of human beings is limited and ultimately not worth centralizing at all, so the EFFICIENT MARKETS HYPOTHESIS is pure dogma.  So without further ado, the code and the graph of the backtest.  The function cstrat2 with parameters thresh=0.0,holding days=1,number of days to calculate correlations=30 produces the results.  There is no cheating (in terms of hidden look-ahead type error) obviously although it’s not trivial to catch such bugs.  This is a real (backtest) result which could be implemented live.  The interesting issue is precisely the issue of efficient market hypothesis.  So how long have people been doing arbitrage with PAIRS?  Well, say from the late 1970s.  So in principle PAIRS should not produce ANY profits especially when the conglomerate (gold mining stocks) has been of interest for a long time.  And yet, a simple variation of basic pairs STILL produces without great deal of sophistication, a sizeable arbitrage opportunity.  I am tempted to conclude that the human capacity for ARBITRAGE is much more limited than is justified by the efficient markets hypotheses.  

library(quantmod)
library(tseries)
 
symbols<-c(“ABX”,”NEM”,”NCM”,”GG”,”FNV”,”AEM”,”SLW”,”GOLD”,”RGLD”,”AU”)
indexName<-“GDX”
 
 
# Get historical data and construct table
 
constructComponentRets<-function(syms,StartDate)
{
Stocks = lapply(syms, function(sym) {
  dailyReturn(na.omit(getSymbols(sym, from=StartDate, auto.assign=FALSE)))
})
Stocks<-do.call(merge, Stocks)
Stocks
}
 
constructIndexRets<-function(indexName,StartDate)
{
    dailyReturn(getSymbols(indexName,from=StartDate,auto.assign=FALSE))
}
 
 
componentRets<-constructComponentRets(symbols,”2007-01-03″)
indexRets<-constructIndexRets(indexName,”2007-01-03″)
 
 
cstrat<-function(thresh,days,corrdays){
    # some best parameters 0.0,4,100
    rets<-componentRets
    rets[is.na(rets)]<-0
    Z<-merge(componentRets[,5],indexRets)
    meanport<-Z[,2]
    meanport[is.na(meanport)]<-0
    N<-nrow(rets)
    v<-rep(0,N)
    dirs<-rep(0,N)
    bprev<-2
 
for (a in (corrdays+1):(N-days-1)){
m<-mean(meanport[(a-10):a])
if (is.na(m)){m=0}
 
# choose the stock with the highest correlation
# in the past 20 days
 
CC<-rep(0,ncol(rets))
for (bb in 1:ncol(rets)){
CC[bb]<-cor( meanport[(a-corrdays):(a-1)],rets[(a-corrdays):(a-1),bb])
}
 
b<-bprev
if (!is.na(sum(CC))){
    b<-which.min(CC)[1]
    bprev<-b
}
else {
    print(CC)
}
R<-mean(rets[(a-10):a,b])
if (is.na(R)){R<-0}
print(paste(R,m))
if (abs(R-m)>thresh){
 
direction<- sign(R-m)
DinWords<-“Long Port Short Stock”
if (direction<0){
DinWords<-“Short Portfolio Long Stock”
}
 
print(paste(symbols[[b]],b,CC[b],DinWords))
 
#tryCatch({
#if (v[a]< 0 && v[a-1]<0 && v[a-2]<0 && dirs[a] == dirs[a-1] && dirs[a-2]==dirs[a-1]){
#direction<- -dirs[a]
#}
#},error=function(e){}
#)
    dirs[a+1]<-direction
    val<-mean(meanport[a+1:days]-rets[a+1:days,b])
    if (is.na(val)){
        val<-0
    }
    v[a+1]<- direction*val
}
}
print(paste(‘SD=’,sd(v),’mean=’,mean(v),’av sd=’,
sd(meanport),’av mean=’,mean(meanport)))
sharpe<-(sum(v)-sum(X[,12]))/max(0.0001,sd(v)*sqrt(254))
print(paste(‘Sharpe=’,sharpe))
stratChoices<-xts(order.by=index(rets))
stratChoices<-merge(stratChoices,v)
    stratChoices<-merge(stratChoices,dirs)
    plot(cumsum(stratChoices[,1]))
stratChoices
 
}
 
 
cointegration<-function(x,y)
{
vals<-data.frame(x,y)
beta<-coef(lm(vals[,2]~vals[,1]+0,data=vals))[1]
(adf.test(vals[,2]-beta*vals[,1], alternative=”stationary”, k=0))$p.value
}
 
cstrat2<-function(thresh,days,corrdays){
    # some best parameters 0.0,4,100
    rets<-componentRets
    rets[is.na(rets)]<-0
    Z<-merge(componentRets[,5],indexRets)
    meanport<-Z[,2]
    meanport[is.na(meanport)]<-0
    N<-nrow(rets)
    v<-rep(0,N)
    dirs<-rep(0,N)
    bMinPrev<-2
    bMaxPrev<-2
for (a in (corrdays+1):(N-days-1)){
m<-mean(meanport[(a-10):a])
if (is.na(m)){m=0}
 
 
# choose the stock with the highest correlation
# in the past 20 days
 
CC<-rep(0,ncol(rets))
for (bb in 1:ncol(rets)){
CC[bb]<-cor( meanport[(a-corrdays):(a-1)],rets[(a-corrdays):(a-1),bb])
}
 
bmin<-bMinPrev
bmax<-bMaxPrev
if (!is.na(sum(CC))){
    bmin<-which.min(CC)[1]
    bMinPrev<-bmin
    bmax<-which.max(CC)[1]
    bMaxPrev<-bmax
}
else {
    print(CC)
}
 
# cointegration p-value
#x<-rets[1:a,bmin]
#y<-rets[1:a,bmax]
#ci.pv<-cointegration(x,y)
#print(paste(‘CIPV=’,ci.pv))
 
Rmin<-mean(rets[(a-2):a,bmin])
Rmax<-mean(rets[(a-2):a,bmax])
if (abs(Rmax-Rmin)>thresh){
 
direction<- sign(Rmax-Rmin)
DinWords<-“Long Port Short Stock”
if (direction<0){
DinWords<-“Short Portfolio Long Stock”
}
 
 
print(paste(‘Max:’,symbols[[bmax]],bmax,CC[bmax],DinWords))
print(paste(‘Min:’,symbols[[bmin]],bmin,CC[bmin],DinWords))
 
#tryCatch({
#if (v[a]< 0 && v[a-1]<0 && v[a-2]<0 && dirs[a] == dirs[a-1] && dirs[a-2]==dirs[#a-1]){
#direction<- -dirs[a]
#}
#},error=function(e){}
#)
    dirs[a+1]<-direction
    val<-mean(rets[a+1:days,bmax]-rets[a+1:days,bmin])
    if (is.na(val)){
        val<-0
    }
    v[a+1]<- direction*val
}
}
print(paste(‘SD=’,sd(v),’mean=’,mean(v),’av sd=’,
sd(meanport),’av mean=’,mean(meanport)))
sharpe<-(sum(v)-sum(X[,12]))/max(0.0001,sd(v)*sqrt(254))
print(paste(‘Sharpe=’,sharpe))
stratChoices<-xts(order.by=index(rets))
stratChoices<-merge(stratChoices,v)
    stratChoices<-merge(stratChoices,dirs)
    plot(cumsum(stratChoices[,1]),main=”Backtest of Gold Pairs Strategy”)
stratChoices
 
}
The output from this code should end as follows in case you want to run it.

[1] “Min: NCM 3 NA Short Portfolio Long Stock”
 [1] 0.7847300 0.8726714        NA 0.7700649 0.8316100 0.8164236 0.8714387 0.8393124 0.8181947 0.8939798
[1] “Max: SLW 7 0.87143867038477 Short Portfolio Long Stock”
[1] “Min: NCM 3 NA Short Portfolio Long Stock”
 [1] 0.7842892 0.8768577        NA 0.7765014 0.8296686 0.8118670 0.8751698 0.8298539 0.8248970 0.8925556
[1] “Max: SLW 7 0.875169814860507 Short Portfolio Long Stock”
[1] “Min: NCM 3 NA Short Portfolio Long Stock”
 [1] 0.7861991 0.8728876        NA 0.7659750 0.8205137 0.8068286 0.8693557 0.8199464 0.8164895 0.8846133
[1] “Max: SLW 7 0.869355689699331 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8183631 0.8665196        NA 0.7701502 0.8430734 0.8178325 0.8775521 0.8059414 0.8265138 0.8878411
[1] “Max: SLW 7 0.877552110562935 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8202531 0.8781273        NA 0.7692949 0.8435192 0.8186440 0.8794814 0.7999895 0.8267055 0.8876754
[1] “Max: SLW 7 0.879481365179698 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8235891 0.8627736        NA 0.7591603 0.8357791 0.8093991 0.8701011 0.7784181 0.8089477 0.8688979
[1] “Max: SLW 7 0.870101141833533 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8759150 0.8951543        NA 0.8318822 0.8822690 0.8754342 0.9111954 0.8150941 0.8711518 0.9181651
[1] “Max: SLW 7 0.911195353619005 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8720164 0.8664675        NA 0.8263342 0.8736644 0.8710797 0.9157766 0.8174924 0.8958259 0.9101778
[1] “Max: SLW 7 0.915776562559851 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8736416 0.8647396        NA 0.8306787 0.8769669 0.8696473 0.9157012 0.8185065 0.9055496 0.9093271
[1] “Max: SLW 7 0.915701227312764 Short Portfolio Long Stock”
[1] “Min: NCM 3 NA Short Portfolio Long Stock”
 [1] 0.8715562 0.8476112        NA 0.8156205 0.8880961 0.8670091 0.9140808 0.8098111 0.8994516 0.9082622
[1] “Max: SLW 7 0.914080812730942 Short Portfolio Long Stock”
[1] “Min: NCM 3 NA Short Portfolio Long Stock”
 [1] 0.8748459 0.8551964        NA 0.8194074 0.8895541 0.8697001 0.9038042 0.8324408 0.9000305 0.9113899
[1] “Max: SLW 7 0.903804164280884 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8734272 0.8527967        NA 0.8157374 0.8725731 0.8668520 0.7595406 0.8312755 0.8945350 0.9084764
[1] “Max: SLW 7 0.759540622320781 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8694111 0.8511296        NA 0.8224220 0.8710239 0.8688679 0.7376991 0.8208379 0.8914138 0.9042604
[1] “Max: SLW 7 0.737699051475897 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8657232 0.8480939        NA 0.8205625 0.8694782 0.8663341 0.7343544 0.8207944 0.8896814 0.8984146
[1] “Max: SLW 7 0.73435444368399 Short Portfolio Long Stock”
[1] “Min: NCM 3 NA Short Portfolio Long Stock”
 [1] 0.8704843 0.8519156        NA 0.8214306 0.8751230 0.8691931 0.7453998 0.8235559 0.8977702 0.9020496
[1] “Max: SLW 7 0.745399845253296 Long Port Short Stock”
[1] “Min: NCM 3 NA Long Port Short Stock”
 [1] 0.8777785 0.8586096        NA 0.7957486 0.8798905 0.8746327 0.7461774 0.8248994 0.8915092 0.9045272
[1] “Max: SLW 7 0.746177356267977 Short Portfolio Long Stock”
[1] “Min: NCM 3 NA Short Portfolio Long Stock”
[1] “SD= 0.0173418135298884 mean= 0.00189100848160369 av sd= 0.0281268089045118 av mean= 0.000189184692073368”
[1] “Sharpe= 15.3981976285237”
2 Attachments

Read Full Post »

Stocks: “ABX” “NEM” “GG” “AEM” “RGLD” “KGC” “GFI” “PVG” “HMY” “RIC”
“GORO”

Before we get to the actual code, the idea is simple.  It’s to calculate the past 100 day correlations of the stocks and do pairs trading on the stock with the maximum correlation and the portfolio with a holding period of 4 days (which gives a Sharpe versus S&P 500 of > 18.0 estimated roughly by dividing sum(strategy return)-sum(SP500 return) by sigma(strategy return)*sqrt(254).  More useful is the graph.  These types of strategies (with good returns) are actually not extremely hard to construct in backtests although they are unstable and finicky.  In this case, if you used past correlations much more or less than 100 will make the performance much worse and almost unusable.  So there is the question of whether there are fundamental laws governing such things.  If there are, they are not connected to concepts of daily correlations.

April-2017-gold-strategy

symbols = c("ABX" "NEM" "GG" "AEM" "RGLD" "KGC" "GFI" "PVG" "HMY" "RIC" 
"GORO","SPY")
getSymbols(symbols)
rets = merge(lapply(symbols, function(s){ dailyReturn(eval(parse(text=s))) })

cstrat<-function(thresh,days,corrdays){
meanport<-rowMeans(rets)
meanport[is.na(meanport)]<-0
N<-nrow(rets)
v<-rep(0,N)
dirs<-rep(0,N)
for (a in (corrdays+1):(N-days-1)){
m<-mean(meanport[(a-10):a])
if (is.nan(m)){m=0}

# choose the stock with the highest correlation
# in the past 20 days

#C<-cor(rets[(a-21):a,])
#C[is.na(C)]<-0
#diag(C)<-0
#idx<-which(C==max(C),arr.ind=TRUE)[1]

CC<-rep(0,ncol(rets))
for (bb in 1:ncol(rets)){
CC[bb]<-cor( meanport[(a-corrdays):a],rets[(a-corrdays):a,bb])
}
#mb<-median(CC)
#CD<-abs(CC-mb)
b<-which.max(CC)
print(paste(b,CC[b]))
R<-mean(rets[(a-10):a,b])
if (abs(R-m)>thresh){
direction<- -sign(R-m)
if (v[a]< 0 && v[a-1]<0 && v[a-2]<0 && dirs[a] == dirs[a-1] && dirs[a-2]==dirs[a-1]){
direction<- -dirs[a]
}
dirs[a+1]<-direction
v[a+1]<- direction*mean(meanport[a+1:days]-rets[a+1:days,b])
}
}
print(paste('SD=',sd(v),'mean=',mean(v),'av sd=',
sd(meanport),'av mean=',mean(meanport)))
plot(cumsum(v))
(sum(v)-sum(X[,12]))/max(0.0001,sd(v)*sqrt(254))
}

 

Read Full Post »

It seems clearer now that the Einstein static universe can remove dark energy and dark matter and solve the cosmological constant problem and explain the CMB anisotropy as well.  Euclidean quantum field theory via Osterwalder-Schrader axioms can be formulated for it by Jaffee-Ritter.  This leaves only the hierarchy problem which is definitely easier to address in this situation.  The hierarchy between gravity and electromagnetism should be a simple function of the radius of the static universe.  More interesting issue is whether my initial (and naive excitement) that quantization itself is a CONSEQUENCE of a static Einstein universe and perhaps determines quantum field theory is true.  So this is a great problem.  While it may seem as naive to say that a static spherical geometry can describe quantum field theory as the pre-Keplerian idea that the orbits of planets must be perfect circles, in this case since Planck’s constant is a universal quantization of ALL energy in the universe and not only the energy that emanates from a particular atom, this is actually a NATURAL CONJECTURE if we rewind the clock back to 1900.  If we return to 1900 armed now with some knowledge that the universe IS an Einstein static universe and showed PLANCK the cosmic background radiation then he might have decided that the entire universe must be a compact blackbody and the Planck’s quantum hypothesis must then apply to the entire universe.  In other words a static Einstein (or S4) universe then justifies the Planck’s constant itself naturally.

Read Full Post »

Older Posts »