5. The Art of Possible
The flexible configuration options and processor logic in gecholog
provide a toolkit to integrate to your specific use of LLMs. With its open architecture for processors, gecholog
allows you to build your own custom processors for your unique requirements. You can write these in any programming language or utilize one of our available processors.
Augment your data
Processors in gecholog
can operate synchronously, enhancing both the request and the response in interactions with the LLM service:
- Add data to your payload (e.g., insert data into your prompts, change settings, etc.).
- Add or modify headers.
- Replace parts of your request or response.
Processors can also function in an asynchronous mode to minimize impact on request latency. Asynchronous, in this context, means post-processing after the response has already been sent back to the user.
- Run external classifiers on requests/responses to include NLP data for improved tracking.
- Implement counters and measurements.
- Parse and modify JSON fields, then add them to the log.
- Create and apply meta-data tags.
Clean your data
In modern architectures, data privacy is a crucial consideration. For gecholog
, you can develop processors to:
- Alter fields before they're forwarded to the target or returned to the requester.
- Obfuscate PII (Personally Identifiable Information) from logs via post-processing.
- Selectively filter fields to be included in the final log.
Measure your traffic
gecholog
consistently records timestamps and measures the duration of various stages:
- Time from ingress to outbound.
- Duration from outbound to inbound.
- Interval from inbound to egress.
- Time taken in post-processing.
- Duration for each processor.
Control traffic
Should you need to establish rules for controlling or blocking traffic before it reaches your target, gecholog
request processors are equipped for the task. You have the capability to:
- Throttle traffic based on token consumption, utilizing the standard library
tokencounter
processor. - Develop a custom content filter tailored to your needs.
- Design a processor that leverages an external API to respond to specific types of requests.
Use Cases
Here are a couple of examples to describe the Art of Possible. And how you can solve them using gecholog
.
PII-cleaner
Remove common patterns like social-security number, phone numbers, addresses etc from all payloads.
Create a processor that parses payloads and replace PII-patterns with *
characters. Configure gecholog
to run the processor async=true
both as a request
and response
processor.
Content filter
Block requests of certain content before sending it to the LLM-service.
Write a processor that uses 3rd party classification service to classify the content according to your rules (for example identifying non-public company documents). If the classification rating is over a certain threshold, write a json response the control
field to block sending traffic to the target. Configure gecholog
to run the request
processor async=false
for the ingress_payload
field. See the Custom Processor contentfilter for inspiration.
Use a cache service
Hook up a cache service to provide answers to common questions where the LLM request is not needed.
Your processor would catch the ingress_payload
and validate if the cache-service has an answer to propose. If the cache-service has an answer, use the control
field send back the response from the cache-service to the user. Configure gecholog
to run the processor async=false
as a request
processor.
Store metrics but no data
You want to track the performance, latency and token cost of your llm-traffic, but you don't want the logs to contain any free-text data.
You don't need a processor to accomplish this. Just configure either the fields_include
or the fields_exclude
settings in the logger
part of the configuration file /app/conf/gl_config.json
. These settings determine what fields will be in the final log.
Augment the prompt
You application is reusing similar requests and you want to decouple part of the prompt building from the requestor.
Build a processor to modify the outbound_payload
and to expand the prompt according to your templates. Configure gecholog
to run your processor as a request
processor with modifier=true
for the field outbound_payload
.
Analyze api calls that belong together
I want to track user sessions, not only each api call.
You don't need a processor to accomplish this, just configure the request from the application to use the built in session_id
feature of gecholog
. Just send the last session_id
in the header for each api call that below together and your logs get the same session_id
but different transaction_id
.
I want to learn more
Recommended reading to dig deeper into the the world of gecholog
- Learn about our Standard Processors
- Master how to configure
gecholog
in section Configuration - Start to explore how to create a Custom Processor