Add Socket connection?

[zoakes]

I've been thinking about adding a socket connection class to a generic FIX IP, was curious if there's anything in beta for this? If not, I'm happy to fork and contribute what I can to it if I have some success -- I was going to connect to IB.

Let me know your thoughts!

Simplefix is INCREDIBLE, so simple -- exactly what I was looking for -- FIX formatting without the overhead mess of QuickFix -- just needs a Socket initiator : )

Zach

[da4089]

Hi Zach,

At one time I was fairly opposed to this idea. I think it's a slippery slope from adding some sort of socket support, and then maybe some sort of message validation, and suddenly we're back where QuickFIX is.

But I'm maybe less against it than I was, mostly because I think a lot of people end up having to write it themselves, and while that's not terribly hard, it does seem like maybe an off-the-shelf way to do it would be better.

So. This is maybe a little unfair, but I'd kinda be interested in seeing what you come up with, without really promising to merge the PR if I get cold feet. But ... since you're writing it anyway, and since you're kinda volunteering, I'm absolutely happy to look at a PR and work with you on merging it if it seems promising?

[zoakes]

Sure. Well mine is very simple, but I’m happy to send. I have it on a git, in the Socket+ branch (with generic connection now) https://github.com/zoakes/sFIX-IB/blob/Socket%2B/SFX_Sock.py Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: David Arnold <notifications@github.com> Sent: Monday, November 25, 2019 5:51:27 AM To: da4089/simplefix <simplefix@noreply.github.com> Cc: Zach Mazz <zach_oakes@outlook.com>; Author <author@noreply.github.com> Subject: Re: [da4089/simplefix] Add Socket connection? (#22) Hi Zach, At one time I was fairly opposed to this idea. I think it's a slippery slope from adding some sort of socket support, and then maybe some sort of message validation, and suddenly we're back where QuickFIX is. But I'm maybe less against it than I was, mostly because I think a lot of people end up having to write it themselves, and while that's not terribly hard, it does seem like maybe an off-the-shelf way to do it would be better. So. This is maybe a little unfair, but I'd kinda be interested in seeing what you come up with, without really promising to merge the PR if I get cold feet. But ... since you're writing it anyway, and since you're kinda volunteering, I'm absolutely happy to look at a PR and work with you on merging it if it seems promising? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fda4089%2Fsimplefix%2Fissues%2F22%3Femail_source%3Dnotifications%26email_token%3DALPUCIEUIL46AETWA7SWCXTQVO337A5CNFSM4JRBVJR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFCEFJY%23issuecomment-558121639&data=02%7C01%7C%7Cdf980eb5c15b4fedf91608d7719dcde3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637102794878485564&sdata=FqC8p1RV3EcWV1Ec%2Fn9Ttc9XFCvtjOHQLpJA%2Fi3DRnY%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALPUCIEKPYLDVTVDRO6QRYLQVO337ANCNFSM4JRBVJRQ&data=02%7C01%7C%7Cdf980eb5c15b4fedf91608d7719dcde3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637102794878495563&sdata=MB9bDfg9aRECiuBRO8igHI5E%2Bh3YvCW%2B86GafyIJqRU%3D&reserved=0>.

[zoakes]

It’s far from QuickFix haha — I like simple too. QF is just so bloated, and their documentation is terrible. I hate QF. Also — I realize I need to remove the finally’s to the very bottom to keep connection open (in sendAll) — just was benchmarking and can’t if it’s running continuously. Goal was the most minimal connection to IB’s FIX protocol. Zach Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Zach Oakes <zach_oakes@outlook.com> Sent: Monday, November 25, 2019 7:36:31 AM To: da4089/simplefix <reply@reply.github.com>; da4089/simplefix <simplefix@noreply.github.com> Cc: Author <author@noreply.github.com> Subject: Re: [da4089/simplefix] Add Socket connection? (#22) Sure. Well mine is very simple, but I’m happy to send. I have it on a git, in the Socket+ branch (with generic connection now) https://github.com/zoakes/sFIX-IB/blob/Socket%2B/SFX_Sock.py Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________ From: David Arnold <notifications@github.com> Sent: Monday, November 25, 2019 5:51:27 AM To: da4089/simplefix <simplefix@noreply.github.com> Cc: Zach Mazz <zach_oakes@outlook.com>; Author <author@noreply.github.com> Subject: Re: [da4089/simplefix] Add Socket connection? (#22) Hi Zach, At one time I was fairly opposed to this idea. I think it's a slippery slope from adding some sort of socket support, and then maybe some sort of message validation, and suddenly we're back where QuickFIX is. But I'm maybe less against it than I was, mostly because I think a lot of people end up having to write it themselves, and while that's not terribly hard, it does seem like maybe an off-the-shelf way to do it would be better. So. This is maybe a little unfair, but I'd kinda be interested in seeing what you come up with, without really promising to merge the PR if I get cold feet. But ... since you're writing it anyway, and since you're kinda volunteering, I'm absolutely happy to look at a PR and work with you on merging it if it seems promising? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fda4089%2Fsimplefix%2Fissues%2F22%3Femail_source%3Dnotifications%26email_token%3DALPUCIEUIL46AETWA7SWCXTQVO337A5CNFSM4JRBVJR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFCEFJY%23issuecomment-558121639&data=02%7C01%7C%7Cdf980eb5c15b4fedf91608d7719dcde3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637102794878485564&sdata=FqC8p1RV3EcWV1Ec%2Fn9Ttc9XFCvtjOHQLpJA%2Fi3DRnY%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALPUCIEKPYLDVTVDRO6QRYLQVO337ANCNFSM4JRBVJRQ&data=02%7C01%7C%7Cdf980eb5c15b4fedf91608d7719dcde3%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637102794878495563&sdata=MB9bDfg9aRECiuBRO8igHI5E%2Bh3YvCW%2B86GafyIJqRU%3D&reserved=0>.

[zoakes]

I added some stuff that most people would probably want -- like heartbeat, logging, etc. I think for now I'm done, this worked for me today in some preliminary testing, so I'm pretty pleased hah. I expected this fix engine to take like.. weeks-months, so thank you for saving me that : )
End goal is Cython / C++ engine, so I'm going to get started on optimizing tomorrow.

I'm happy to work with you on the PR (pull request?) -- I'm not super familiar with all the github ins and outs, but I'm a solid developer. If it's beginning to feel a bit like QF, no need to merge -- I'm indifferent, really I'm happy I have that makes sense and works well in a few hundred lines. Let me know !

Thank you again for this incredible module.

Zach

[zoakes]

David,

I finished a really clean Python FIX client with simplefix -- I'll attach the current version here. I finally finished up integrating / testing with Interactive Brokers IB CTCI FIX API, they are incredibly particular about what's used, what isn't, what's in header, etc -- but it's ready now!
If you'd like to use it, or pull it -- you're welcome to.
I currently have it running the heartbeat as a Daemon thread, but I have a few versions of heartbeat that can be used -- this has been the most consistent.

https://github.com/zoakes/sFIX-IB/blob/master/base_hb_v2.0.py

I didn't add any message validation, I'm not quite sure how to aside from just using a listener -- which sounds bloated. If it is something you're interested in I can give it a try though.

[mecanda12]

@da4089 @zoakes Hey David,

If Zach, code is worth it, is any possibility to merge it? I see his code help connect to the backend engine.

[da4089]

I will definitely have a look at this. I'm still unsure how I'd like to integrate it, but at a minimum, it could be in some sort of example code directory. The next couple of weeks are a little busy, but I should get a chance to look at it after that.

Thank you for this!

[mecanda12]

Thanks, David :)

[zoakes]

My code was pretty minimal — more scaffolding to build out your own spec with if I recall. Honestly, there’s a lot of FIX overhead in maintaining connectivity, and with the new FIX5 / SP stuff (session and transport layer split), it’s even more substantial. I can run over it again, but I didn’t intend this to be merged into main. If you find use in it, go for it of course :) I’m guessing something generic that applies + passes cert at various FIX endpoints would be substantially larger — and not really fit with the idea here. We run FIX at CME on ILink3 and it’s like thousands of lines of code to maintain connectivity, persist messages for reconnect / disconnect / reconciling msg counts, etc. it’s not small. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: David Arnold ***@***.***> Sent: Wednesday, April 5, 2023 9:07:19 AM To: da4089/simplefix ***@***.***> Cc: Zach Mazz ***@***.***>; Mention ***@***.***> Subject: Re: [da4089/simplefix] Add Socket connection? (#22) I will definitely have a look at this. I'm still unsure how I'd like to integrate it, but at a minimum, it could be in some sort of example code directory. The next couple of weeks are a little busy, but I should get a chance to look at it after that. Thank you for this! — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALPUCIEQVKBBXHKGDIP4UUTW7V4BPANCNFSM4JRBVJRQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[da4089]

For better or worse, there's not a lot of usage of the new session layer stuff that I've seen: most FIX is still using the classic session protocol and some variant of FIX4.x.

But yes, a production quality FIX engine is a heap of work. In most cases, you'd probably be better off starting with the Python QuickFIX or WTFIX if that's what you need.

That said, something that's reasonably resilient and suitable for running regression tests, etc, isn't too much effort.

[mecanda12]

@da4089 I am looking for a solution for a Non-production environment. Most a Test environment.
My question is the same how can I make a network connection using simplefix, QuickFIX, or WTFIX?

[zoakes]

Yep. Even then -- quickfix doesn’t handle the new ‘split layer’ ILink3 stuff, or doesn’t publish anything implying they can. We ended up building one, but it took months. For basic stuff, sure – could be fairly minimal. Just a heartbeat, some other minimal overhead. If you want something to truly manage a prod level OMS / MD connection – it will quickly look like QuickFix. If that was the goal, I’d say this probably deviates from the ‘simple’ goal of the library. Also of note – many of the paid FIX engine players barely have FIX5 / SP stuff working / tested. CME is shifting entirely to ILink3 in 2024, phasing out ILink2 now (FIX4), but no new connections can use FIX4 for example. It’s a bit of a unique time in FIX software where everyone w DMA is preparing to scramble or break. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows From: David ***@***.***> Sent: Wednesday, April 5, 2023 9:29 AM To: ***@***.***> Cc: Zach ***@***.***>; ***@***.***> Subject: Re: [da4089/simplefix] Add Socket connection? (#22) For better or worse, there's not a lot of usage of the new session layer stuff that I've seen: most FIX is still using the classic session protocol and some variant of FIX4.x. But yes, a production quality FIX engine is a heap of work. In most cases, you'd probably be better off starting with the Python QuickFIX or WTFIX if that's what you need. That said, something that's reasonably resilient and suitable for running regression tests, etc, isn't too much effort. — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALPUCICFLYG67APDUWZMB3LW7V6VBANCNFSM4JRBVJRQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[zoakes]

How can you make a network connection… as in how do you handle it from a software perspective? or how do you get the connectivity? Software is fairly simple – the connectivity maybe not. DMA will run in the hundreds of thousands. Some brokers offer FIX alternatives to REST stuff, like IB, but that’s sort of FIX – lite. Just depends, but you’ll likely need to bring some $ to the table for anyone to certify with you. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows From: Zach ***@***.***> Sent: Wednesday, April 5, 2023 11:23 AM To: ***@***.***>; ***@***.***> Cc: ***@***.***> Subject: RE: [da4089/simplefix] Add Socket connection? (#22) Yep. Even then -- quickfix doesn’t handle the new ‘split layer’ ILink3 stuff, or doesn’t publish anything implying they can. We ended up building one, but it took months. For basic stuff, sure – could be fairly minimal. Just a heartbeat, some other minimal overhead. If you want something to truly manage a prod level OMS / MD connection – it will quickly look like QuickFix. If that was the goal, I’d say this probably deviates from the ‘simple’ goal of the library. Also of note – many of the paid FIX engine players barely have FIX5 / SP stuff working / tested. CME is shifting entirely to ILink3 in 2024, phasing out ILink2 now (FIX4), but no new connections can use FIX4 for example. It’s a bit of a unique time in FIX software where everyone w DMA is preparing to scramble or break. Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows From: David ***@***.***> Sent: Wednesday, April 5, 2023 9:29 AM To: ***@***.***> Cc: Zach ***@***.***>; ***@***.***> Subject: Re: [da4089/simplefix] Add Socket connection? (#22) For better or worse, there's not a lot of usage of the new session layer stuff that I've seen: most FIX is still using the classic session protocol and some variant of FIX4.x. But yes, a production quality FIX engine is a heap of work. In most cases, you'd probably be better off starting with the Python QuickFIX or WTFIX if that's what you need. That said, something that's reasonably resilient and suitable for running regression tests, etc, isn't too much effort. — Reply to this email directly, view it on GitHub<#22 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALPUCICFLYG67APDUWZMB3LW7V6VBANCNFSM4JRBVJRQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mecanda12]

@zoakes To make a connection with the backend engine, a socket can be used to communicate with the bundle.

Ip address, Port, Target & Sender comp Id, Target Sub ID, User address and password.

The above information can be used to make a network connection.