/alg/ - Algorithmic Trading General

Also just to add to this, you might be tempted to just throw a machine learning algorithm at it and start plain forecasting. Don't, as this is usually the wrong approach when you start out. Or in general when working on this type of problem. Start with something more traditional like time series anlaysis, applying simple linear models and start thinking in terms of exploitation rather than forecasting, as forecasting is a much harder problem.

Thanks, I had tried to learn some machine learning over the last two years, but a lot of it was going over my head. Maybe with a reason to learn it I'll be more motivated. I'll look at that and the Ernie Chang books, I need a break from Veeky Forums.

Its def much easier to learn something when learning with application in mind. Actually now that i think of it, fast.ai is a great general machine learning/deep learning course. Not finance specific, but all of it can be applied to markets eitherway.

How do you minimize transaction costs when doing algorithmic trading? Or do you make enough to keep the costs at minimum?

I guess you could focus on market making strategies for example as most exchanges will offer you reduced fees for the added liquidity. I don't in general so my backtesting process goes something like this
> Test strat without fees or slippage. Bin it if sharpe or drawdown is unfavorable.
> Test strat with fees but without slippage. Bin it if sharpe or drawdown is unfavorable.
> Finally test strat with fees and with slippage.

Okay thanks, sounds like a lot of tweaking needs to be done before your code would be market ready since the competition is probably rather tough. What trading/broker portal do you use?

Do you have any good resources or recommendations for programming these kind of trading bots in python? Or just financial modeling/analysing with python in general?

I have some experience of data scraping (web scraping historical data from yahoo finance, for example) and I'm now going through a tutorial which uses ML to "predict" or read that data.

Here is a qt for your effort.

qt much appreciated
yes a lot of fuckery is involved in general... A lot of it is very specific to finance too. The data is non-stationary for example and this can lead to weird complications like your model doing well on test/unseen data and then completely fucking off in a real market setting because the dynamics just aren't the same anymore or you were just doing curve fitting all this time.

Biases are hard to deal with and this is why most actual quants recommend focusing on linear models first. For example machine learning and non-linear models in general (except for trees, random forests, etc) require some kind of data scaling/normalization and this leads to very ugly lookahead biases when plain normalization or scaling techniques are applied. 8/10 academic papers commit these type of mistakes and invalidate their models outright.

The competition, comparatively speaking, aint all that desu. Probably partially due to institutional money mostly being uninterested as of now. I used to algotrade options and stocks (not anymore, only focus on smart beta there right now) and finding alpha was tough as fuck. Much easier with crypto.

I usually trade the main pairs on binance and bitfinex directly via their websocket api. Give me a few on that book question, I'll see what I can remember.

to expand on that "competition" point real quick. In crypto even a simple sentiment/news based bot should do rather well. To my knowledge the edge is still there and the alpha hasn't been arbitraged away. In other markets your sentiment bot will probably be exploited by other bots real fucking fast.

I am an engineer, going to take the fast.ai course this coming week. Where do I go from there to learn more about specific algorithms applicable to this?

Is it just basic linear/non-linear models like we learnt in school?

Also what about taxes, how do I report all the crypto to crypto shit if I'm making thousands of transactions?

alright so books.
I don't know your background of course, but in general, I'm gonna assume some knowledge of stats and probability at least. If you don't have that, pick up any good text book and start there.
Read "Trading and Exchanges" by Harris, you wanna understand what it is you are modeling first.

As a start "Python for finance" looks decent enough if you are just starting out (disclaimer: haven 't read it, table of contents looks pretty decent though). In general Bayesian modeling is where its at, so for python you probably wanna focus on pymc3 and here's a free book github.com/markdregan/Bayesian-Modelling-in-Python has one chapter on heteroskedastic volatility, concepts can be applied freely though. Check out this blog and his books too: quantatrisk.com should give you more ideas.

Pure theory:
"Options, Futures, and Other Derivatives" hull (heavy, but probably required reading)
"Quantitative equity portfolio management" Qian

Not him but I found this really retarded bot doing wash trades on HitBTC. I'd put in a tiny super low sell order and it'd lower its sell position below mine. Then I'd time it and place my buy order higher than the bot's just before it sold to itself and it'd sell me its wash trade attempts. Then I'd put in a tiny high buy order and it would raise its buy position. After trading fees I was getting about 0.3% each time. It maintained a maximum spread of 0.5% and seemed to be following the prices of another exchange.

> Where do I go from there to learn more about specific algorithms applicable to this
Anything you can think of can be applied here. Maybe you have an algo that does something with orderbooks. Maybe you have a sentiment based algorithm. Maybe you just look at derivative variables like volatility, current hash processing time, current mining fees, etc and make your forecast based on that. Maybe you perform plain time-series analysis, identify strong serial correlation and use Bollinger Bands to trade mean reversion. The best resources for potential strategies are forex forums, academic papers, etc. None of it will work outright and you will have to tweak and experiment a lot, but thats a given.

> Is it just basic linear/non-linear models like we learnt in school?
Much more than that. For example, people applying a "dumb" machine learning algo on prices alone end up with something like this stackoverflow.com/questions/27289719/time-shift-between-target-and-simulation-output-using-neural-network

All thats happening here is you end up with a "naive predictor" or an identity model. Because you are essentially forecasting a random walk by its past values the model learns that the most likely approximation of t+1 is t+minor noise. You have to understand what this is happening and what you can do about it.

> Also what about taxes, how do I report all the crypto to crypto shit if I'm making thousands of transactions?
Can't help you there. Only advice I can give is, interview a couple of accountants preferably those experienced with trading/crypto and go from there.

>I'm gonna assume some knowledge of stats and probability at least
No worries, I have a technical background but in a rather different field than economics or finance, shouldn't be that difficult to catch up on the mathematics.

Thanks a lot of the book recommendations, man, this is going to keep me busy for a while. I have yet to still finish reading the investing classics...

I'll be monitoring the thread. Keep up the good work. (qt is Michele Alves for anyone interested)

Noticed this on binance as well. Tempted to write something to try and exploit these bots.

OP, I've been working on an arbitrage bot for a while, just working through the kinks at the moment, but I'm terrified that real market conditions and withdrawal waiting times will make it not viable. Have you any experience there? Is this a mistake?

Also this is a great thread, so I'll contribute something. Pic related is just a little something primitive in Python to be used with backtesting. Takes trade results in terms of % profit/loss only, then stores them in an array. There's a lot going on but in a nutshell it's basically iterating randomly through the list of results, and simulating them as if they'd happened. After N trades, the predicted portfolio balance is attained and recorded. This process is repeated M times and the results of this are what you see on the histogram on the bottom.
The one on the top is just a visual representation of the "trades" results to use this backtester on, not even necessary to plot but sure look.

No experience with arbitrage to speak of, sorry. At least in the theory the money is there for sure. Why would you need withdrawals to begin with though? Wouldn't you just allocate money to some exchanges and trade those balances? I understand the risk there of course but if withdrawals are a requirement (if you actually need to shift money around) then yeah, I would reconsider... How are you backtesting this in general?

> Pic related is just a little something primitive in Python
Is this just monte carlo sampling essentially?

>Is this just monte carlo sampling essentially?
Yes

I should add, this backtesting script was made a while ago when I was unsure what route of algo I wanted to go in, and obviously since you need orderbooks with arbitrage, you can't backtest. With arbitrage to test its effectiveness irl you actually have to let it run, which is fairly shit.

Also,
>Wouldn't you just allocate money to some exchanges and trade those balances?
No, from what I've noticed, some exchange will have higher prices than others pretty much all the time, so it nearly always just flows one way. Let's say I buy cheap XMR on Exchange 1 and sell expensive XMR on Exchange 2. On Exchange 2 it's consistently at a higher price, and now I have no more Monero left there to sell. I'd basically be left with only BTC and ETH on the expensive exchange. Look though, work in progress, I intend to incorporate more exchanges so hopefully I wont need to withdraw like you said

> some exchange will have higher prices than others pretty much all the time
Hmm haven't considered that, pretty surprising how the market is still this inefficient.

What if you treat it as a re-balancing problem? So lets say instead of doing on demand allocations/withdrawals you try and minimize your operations in advance. Depending on how many coins are traded and liquidity in general maybe you could even try and make some forecasts? With enough datapoints you should be able to at least somewhat approximate needed allocations. Again, just an idea, no real experience with this.

What's a good exchange with low trading fees for poorfags?

This is December 17th till now and the fees are absolutely murdering its gains.

right now binance

Testing an arbitrage bot on binance - anyone had any success with something similar?

This is my first bot, any tips on minimizing the time between my trades?

yeah Id second binance

I'm dying to get into this but I'm usually so tired from work I simply don't have the energy.

alright boys, gotta head back to trading now. interest was modest but its a start, keep this thread going, Im gonna check back in if its still alive. Otherwise start a new one tomorrow if you feel like it, I'll share a few algo ideas/code, probably setup a github or something we'll see. move it along to something more practical.

user I just want to thank you, I'm going to be doing A LOT of reading this week. This problem has caught my interest and I shall create an excellent bot to compete with yours!