Reverse engineering
Reverse engineering starts with a notebook, not a guess
An evidence-first approach to undocumented protocols: controlled observations, stream boundaries, state machines and fixtures that challenge your assumptions.
Reverse engineering an undocumented network protocol starts by separating observations from explanations. Capture a controlled interaction, record what changed and build a small hypothesis that another experiment can disprove. A parser comes later, once message boundaries and conversation state have enough evidence behind them to survive more than one successful exchange.
Quick answer
Start with a client and environment you are authorized to inspect. Change one input at a time, preserve the raw observations and write down both what you know and what you are only assuming. Keep transport reconstruction separate from application messages. Turn each confirmed behavior into a fixture before expanding the implementation.
yumeko, in the selected work, is a reverse-engineered Yahoo Messenger server implementation. That project is useful context for this niche, but the examples below are a general method. They are not claims about YMSG field layouts or a reconstruction of its original implementation notes.
Make a repeatable observation
A useful capture has a question attached to it. “What happens during login?” is a starting point, but it includes many possible changes: account state, local caches, server negotiation, timing and optional features. A narrower question might ask which bytes differ when a known field changes while the other inputs stay fixed.
Record the application version, the relevant configuration, the exact action and the visible result. Preserve enough context to repeat the experiment. A screenshot of a hexadecimal panel may be convenient for discussion, but the underlying bytes are the fixture. Without them, a later parser test cannot determine whether the explanation still matches the evidence.
Keep a small observation log with three separate fields: the observation, a hypothesis and the next discriminating experiment. For example, observing a two-byte difference near the beginning of several messages does not establish that it is a length. The next experiment could vary payload size while keeping the operation unchanged. That is a proposed experiment, not a conclusion.
A second person should be able to read the log without inheriting every assumption from the first. This is one reason to keep the original capture even after a more readable decoded representation exists.
Reconstruct the transport before naming messages
A stream-oriented transport creates an easy trap: the chunks returned by a read operation can look like application messages in one capture. A later exchange can divide or combine those chunks differently. TCP is specified as a byte-stream transport in RFC 9293; application framing must be established above that layer.
That distinction changes the test corpus. A decoder should be tested with the same recorded conversation delivered in several chunk sizes. A message that decoded correctly when passed as one buffer should still decode when the header arrives separately from its body. Two messages arriving in one buffer should not turn into one invented message.
Wireshark's stream-following view helps inspect a conversation as a stream rather than only as a packet list. It is a useful observation tool, but its display does not identify the application's semantic boundaries for you. Treat the reconstructed bytes as input to the framing investigation.
This separation also helps localize a failure. If the reconstructed stream is incomplete, a new field interpretation is unlikely to repair it. If the bytes are complete but the decoder only works on one chunking, the framing implementation needs attention before the higher-level operation does.
Let the fixtures challenge the parser
A good first decoder has a deliberately small scope. It can recognize a confirmed header, hold incomplete input and return a known message without trying to interpret every field. The first implementation should make uncertainty visible rather than convert every unexplained byte into a confident name.
Suppose an illustrative protocol has a fixed-width length field. The parser still needs a rule for the minimum header size, a maximum accepted body size and behavior when a body is incomplete. Those are design obligations in the example. They do not establish that the real protocol uses that framing scheme.
Keep a raw fixture beside the expected decoded representation. Add a second fixture that contradicts the most convenient assumption: an empty value, an unknown operation, a repeated field or a payload split at an awkward boundary. Whether those cases are legal depends on the evidence. A decoder should not silently decide that unusual input is valid merely because it can be parsed.
Preserve unknown fields where practical. That gives a later interpretation something to compare against, rather than requiring another capture because the first parser discarded everything it did not understand. Also keep rejection behavior explicit. A parser that cannot explain a frame should produce an inspectable failure, not a partly initialized operation that continues into the rest of the application.
Model the conversation as well as the bytes
Correctly naming a message is only one part of compatibility. A client may expect a particular order, a response identifier, an acknowledgement or a state transition before the next action becomes available. Replaying isolated messages can miss those relationships.
Describe the smallest conversation as a state machine. Give the initial state a name, record the action that moves it forward and identify the evidence for the resulting state. If the same bytes behave differently after authentication, that context belongs in the model instead of being hidden in a special case deep inside the decoder.
A stateful test can replay a short exchange from a known starting point. Another can send a valid message in an unexpected state and verify the observed response. A third can interrupt the exchange and establish what a new connection is expected to do. Each test should be tied to an observation or clearly labeled as an implementation policy where the original behavior remains unknown.
Compatibility work benefits from a narrow milestone. Supporting one complete, repeatable conversation is easier to assess than claiming broad support because a long list of operation names exists in a source file. The milestone should say which client version, starting state and user action have been demonstrated.
Write down what remains unknown
An undocumented protocol rarely becomes completely understood in a single pass. The useful deliverable is therefore more than a server that appears to work. It includes the evidence behind that behavior and an honest account of the remaining gaps.
A compact compatibility note can distinguish supported conversations, recognized but unimplemented operations and fields whose meaning is still uncertain. That note gives future experiments a direction. It also prevents a convenient guess from hardening into a public promise when the code is reused.
The same discipline helps when an implementation is rewritten in another language. The fixture corpus and conversation tests become a common reference, while the memory model, I/O layer and parser representation can change underneath. The rewrite has a concrete target beyond reproducing the shape of the previous source code.
From protocol investigation to integration work
My archived yumeko project is part of my independent reverse-engineering work. At Legacies, where I am co-founder and Technical Lead, my responsibilities cover backend architecture, integrations and product stability. Those are separate pieces of my background, connected by the need to understand system boundaries before depending on them.
For someone planning an integration, the useful questions are concrete: what behavior is documented, what can be reproduced and what remains an assumption? Those questions belong in a technical brief before a team estimates or implements the connection. Legacies builds custom applications and automations alongside its own products; the portfolio’s Legacies section explains my role and links to the company.
Frequently Asked Questions
Should I write the parser before collecting several captures?
A small exploratory decoder can help inspect bytes, but it should not become the specification by accident. Preserve multiple controlled observations and treat the early parser as a tool for testing hypotheses rather than proof of the protocol's structure.
Can packet boundaries be used as message boundaries?
Not for an application protocol carried over a TCP byte stream without additional evidence about its framing. The receiving application must be able to handle its input arriving in different chunks while reconstructing the intended messages.
How much of a protocol needs to be understood before testing a server?
A narrow, complete conversation can be a useful milestone. Define the initial state, client version, supported action and expected result so that success is repeatable and does not imply that unrelated features already work.
What should an implementation do with unknown fields?
Where the format permits it, preserving them can support later analysis and compatibility work. Acceptance still needs explicit bounds and error handling; an unknown field is not a reason to trust an arbitrary length or continue with an invalid state.