Element A & Element B: Problem Justification and Past Solution Review

[code259]

Element A & B

Links

• https://sites.google.com/view/longbearscapstone/

Ideas

Communication:

Research Goal: Reduce dependence on paid AI subscriptions enabling student development and growth.
Solution: Host & Serve an LLM (Qwen3.8 27B) locally, so we can get frontier level intelligence with no cost except electricity.

Team split:

• Team 1: Nikhil, Adi & Mihir | Working on Inference Engineering and API Deployment for Rig 1
• Team 2: Yash Squared & Anvay | Working on building Rig 2

Justification:

Problem (LLM subscriptions are costly, and not everyone in OCS has access to frontier intelligence which means their learning and growth is mitigated)
--> Setup GPUs
--> Setup Qwen3.8 27B locally (#4)
--> Means the server can run the LLM without paying API costs; only cost is electricity
--> create API to make the model accesible even without local rig access (#5)
--> detail setup instruction so OCS students can use the model intelligence through a harness like openrouter, pi, or claude code
--> students can use a model harness on their computer which calls the API we made which then runs the prompt they ask on our rig to answer they questions
--> increases their engineering throughput and learning

Literature Review:

• Speculative Decoding: https://arxiv.org/pdf/2211.17192
• YaRN: https://arxiv.org/pdf/2309.00071
• Qwen3.8 27B Model Card: https://huggingface.co/Qwen/Qwen3.8-27B
• TD-Pipe: Temporally-Disaggregated Pipeline Parallelism Architecture for High-Throughput LLM Inference: https://arxiv.org/pdf/2506.10470
• PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers: https://arxiv.org/pdf/2605.02189
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU: https://arxiv.org/pdf/2303.06865
• HeteGen: Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices: https://arxiv.org/pdf/2403.01164
• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU: https://arxiv.org/pdf/2312.12456
• Efficient Memory Management for Large Language Model Serving with PagedAttention / vLLM: https://arxiv.org/pdf/2309.06180
• Orca: A Distributed Serving System for Transformer-Based Generative Models: https://www.usenix.org/system/files/osdi22-yu.pdf
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration: https://arxiv.org/pdf/2306.00978
• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: https://arxiv.org/pdf/2210.17323

1. Host & Serve an LLM

• While it seems pretty simple, serving and managing proper inference workloads is much harder. The field of inference engineering exists for a reason. We also have a unique hardware scenario, so we will have to work with that.
• model → kernels/runtime → inference engine → node configuration → distributed serving → scheduler → networking → autoscaling → observability → reliability → cost optimization
• even for the earlier parts of the stack like kernel & runtime that have good tooling like flash attention and vLLM there is still optimizations you can do based on your workload and usecase
• potential model to run: Qwen3.8 27B (equivalent to GPt 5.6 luna on max; https://artificialanalysis.ai/models/qwen3-8-27b)
• in our case serving that model is nontrivial because of our hardware constraints, so it would be a good learning experience. also, the model is actually good, so people in OCS can use it for programming tasks (would likely be like a subagent with a larger orchestrator like opus, fable, gpt 5.6 sol, or something would delegate to), so would need some interface for people to access the model we serve in their harnesses

Metrics to Track

• TTFT (Time to first token)
• TPS (tokens per second)

Other Ideas

• Make RL envs or use the OCS github/codebase to finetune/improve the model for our usecase.

[code259]

Notes from presentations:

• Add a problem statement with no implementation
• The research questions can include some information on the implementation
• Add an image after that
• for each research paper make a table like that with a one sentence summary
• create phases (if necessary); don't make unnecessary phases - ideally have like 3 phases
• research questions will evolve with the projects
• have research endpoints

[code259]

• "Low cost system for student community, so provide some of the key services like copilot, etc."; talk about the impact statement
• Don't get locked into one model
• Rename: OCS Intelligence LLM
• If want can be a production rig and one be a testing rig; don't want to break the machine on the production instance
  • The other rig can be a testing rig
• Rephrase the "Why this challenging part"
• Delineate Phase 1 and Phase 2 - what each phase is
• Add in parentheses stuff (of the YarN)
• Will grade us after redraft.
• how will we communicate with them (microblogs, etc.)
• make priorities with them

[jm1021]

There are many parts of this system/pipeline. Ultimately, we will always leave one rig running and experiment on the other. 1.) User Experience, 2.) API endpoints, 3.) Broker / Caching work done by Transformers (inputs/outputs) 4.) Determining GPU allocations for Inference Models. 5.) EC2 API to Broker/GPU transportaion/security (Netbird). We may want to have teams more split up on pipline components versus rigs.