M0: compilable skeleton — Kigi 0.1.0 fork surgery
Hard fork of xai-org/grok-build (Apache-2.0) re-targeted as Kigi, an
unofficial Kimi Code CLI community build.
Rename & identity
- 72 xai-*/xai-grok-* crates -> kigi-* (explicit: xai-grok-pager-bin ->
kigi-bin [binary `kigi`], xai-grok-pager -> kigi-tui; rest mechanical);
ptyctl, ptyctl-cli, third_party/ unchanged; proto package
xai.grok.tools.v1 -> kigi.tools.v1
- Config home ~/.kigi (KIGI_SHARE_DIR override), env prefix GROK_* ->
KIGI_*, `kigi --version` carries the unofficial-community-build notice
- clap identity, help text, startup banner, prompt templates rebranded
(templates re-encrypted)
Deletions (PRD removal list #5/#6/#7/#9/#10)
- voice input (xai-grok-voice) and all TUI wiring
- telemetry: Mixpanel client, external OTel stream, Sentry, OTLP layers,
trace/GCS/S3 upload queues (kigi-file-utils halved), workspace upload
module & dc_log, heap-profile uploader, auth-diagnostics uploader,
session-analytics halves of feedback; local zero-egress observability
preserved in new kigi-log crate (unified log, --debug firehose,
subsystem file logs, opt-in instrumentation)
- announcements (crate, remote-settings fields, TUI surfaces)
- plugin marketplace (crate, sources/browse/CTA/extensions-modal tab);
direct plugin install/uninstall/update via kigi-agent git_install kept
- relay/gateway/assets endpoints and features (agent relay, headless
relay transport, gateway bridge, LeaderEnvUrls); leader IPC socket now
~/.kigi/leader.sock + KIGI_LEADER_SOCKET, no ws-url derivation
- functional types rehomed instead of deleted: PermissionMode ->
kigi-config-types, McpInitStrategy -> kigi-mcp, PrCreationSource ->
session signals, TerminalDiagnostics -> kigi-pager-render, agent_id ->
shell util
Endpoints
- kigi-env rewritten: single production KigiEndpoints {coding_api_base_url
https://api.kimi.com/coding/v1 (KIGI_CODE_BASE_URL), oauth_host
https://auth.kimi.com (KIGI_OAUTH_HOST), update_base_url (GitHub
Releases API), upgrade_page_url}; GrokBuildEnvironment enum deleted
Toolchain & workspace hygiene
- Rust 1.97.0 pinned; edition 2024; full cargo update; git2 hoisted to
workspace at 0.21 (Option->Result API migration), quick-xml 0.41
- Root Cargo.toml hand-maintained (PRD §8.1): version 0.1.0 inherited by
all members, members sorted, unused deps pruned
- cargo-deny advisories gate (deny.toml with documented transitive
exceptions); CI workflow (check/clippy/fmt/deny/test, macOS+Linux)
- cross-crate test seams re-gated behind `test-support` cargo feature;
insta snapshot baselines renamed to the kigi_tui prefix
- clippy --workspace --all-targets: zero warnings; fmt clean
Fixes surfaced by the port
- updater probe/installer divergence (bin/kigi vs bin/grok symlink set)
- idle model-metadata refresh dead under KIGI_CODE_BASE_URL override
(new is_effective_coding_endpoint_url, loopback+override aware)
- macOS symlinked-TMPDIR fixture canonicalization (foreign_sessions,
fast-worktree); RSS measurement tests serialized via serial_test
Docs & legal (Apache §4)
- NOTICE added (upstream attribution + change statement); THIRD-PARTY
notices sustained; kigi-tools ported-code notices extended; README,
CONTRIBUTING, SECURITY, AGENTS.md rewritten
Out of scope for M0 (tracked): Kimi auth/inference (M1), search/fetch,
command parity, config import (M2), Computer Hub excision & final
brand-token sweep (M2), distribution & self-update rewrite (M3).
This commit is contained in:
@@ -0,0 +1,279 @@
|
||||
//! Query expansion for FTS-only search mode.
|
||||
//!
|
||||
//! When users ask conversational queries like *"that thing we discussed about the API"*,
|
||||
//! FTS5 matches every word equally — articles, pronouns, and vague references dilute
|
||||
//! precision. This module extracts meaningful keywords by removing stop words.
|
||||
//!
|
||||
//! The pipeline:
|
||||
//! ```text
|
||||
//! query → lowercase → split on non-alphanumeric → remove stop words → dedup → keywords
|
||||
//! ```
|
||||
//!
|
||||
//! When all words are stop words (e.g. "what is that?"), returns an empty vec.
|
||||
//! The caller (hybrid search) falls back to the vector path in that case.
|
||||
|
||||
use std::collections::HashSet;
|
||||
use std::sync::LazyLock;
|
||||
|
||||
static STOP_WORDS: LazyLock<HashSet<&'static str>> = LazyLock::new(|| {
|
||||
[
|
||||
// Articles & determiners
|
||||
"a",
|
||||
"an",
|
||||
"the",
|
||||
"this",
|
||||
"that",
|
||||
"these",
|
||||
"those",
|
||||
// Pronouns
|
||||
"i",
|
||||
"me",
|
||||
"my",
|
||||
"we",
|
||||
"our",
|
||||
"you",
|
||||
"your",
|
||||
"he",
|
||||
"she",
|
||||
"it",
|
||||
"they",
|
||||
"him",
|
||||
"her",
|
||||
"its",
|
||||
"them",
|
||||
"us",
|
||||
// Common verbs
|
||||
"is",
|
||||
"are",
|
||||
"was",
|
||||
"were",
|
||||
"be",
|
||||
"been",
|
||||
"being",
|
||||
"have",
|
||||
"has",
|
||||
"had",
|
||||
"do",
|
||||
"does",
|
||||
"did",
|
||||
"will",
|
||||
"would",
|
||||
"could",
|
||||
"should",
|
||||
"can",
|
||||
"may",
|
||||
"might",
|
||||
// Prepositions
|
||||
"in",
|
||||
"on",
|
||||
"at",
|
||||
"to",
|
||||
"for",
|
||||
"of",
|
||||
"with",
|
||||
"by",
|
||||
"from",
|
||||
"about",
|
||||
"into",
|
||||
"through",
|
||||
"during",
|
||||
"before",
|
||||
"after",
|
||||
"above",
|
||||
"below",
|
||||
// Conjunctions
|
||||
"and",
|
||||
"or",
|
||||
"but",
|
||||
"if",
|
||||
"then",
|
||||
"because",
|
||||
"as",
|
||||
"while",
|
||||
"when",
|
||||
"where",
|
||||
"what",
|
||||
"which",
|
||||
"who",
|
||||
"how",
|
||||
"why",
|
||||
// Vague references
|
||||
"thing",
|
||||
"things",
|
||||
"stuff",
|
||||
"something",
|
||||
"anything",
|
||||
"everything",
|
||||
"one",
|
||||
"some",
|
||||
"any",
|
||||
"all",
|
||||
"each",
|
||||
"every",
|
||||
"both",
|
||||
"few",
|
||||
"more",
|
||||
// Time references
|
||||
"yesterday",
|
||||
"today",
|
||||
"tomorrow",
|
||||
"earlier",
|
||||
"later",
|
||||
"recently",
|
||||
"now",
|
||||
"just",
|
||||
"already",
|
||||
"still",
|
||||
"yet",
|
||||
// Request words
|
||||
"please",
|
||||
"help",
|
||||
"find",
|
||||
"show",
|
||||
"get",
|
||||
"tell",
|
||||
"give",
|
||||
"make",
|
||||
// Common filler
|
||||
"not",
|
||||
"no",
|
||||
"yes",
|
||||
"also",
|
||||
"too",
|
||||
"very",
|
||||
"really",
|
||||
"here",
|
||||
"there",
|
||||
"so",
|
||||
"up",
|
||||
"out",
|
||||
"like",
|
||||
"than",
|
||||
"other",
|
||||
"only",
|
||||
]
|
||||
.into_iter()
|
||||
.collect()
|
||||
});
|
||||
|
||||
/// Extract meaningful keywords from a conversational query by removing stop words.
|
||||
///
|
||||
/// Returns keywords in order of appearance, deduplicated. Words shorter than
|
||||
/// 2 characters and pure-numeric tokens are filtered out. The 2-char minimum
|
||||
/// preserves meaningful short terms like "go", "js", "ui", "db", "ai", "ml"
|
||||
/// while stop words handle the common 2-letter noise ("is", "it", "do", "we").
|
||||
///
|
||||
/// Returns an empty vec when all words are stop words or the query contains
|
||||
/// no meaningful content — the caller should fall back to vector search.
|
||||
pub fn extract_keywords(query: &str) -> Vec<String> {
|
||||
let lowered = query.to_lowercase();
|
||||
let mut seen = HashSet::new();
|
||||
lowered
|
||||
.split(|c: char| !c.is_alphanumeric() && c != '_')
|
||||
.filter(|w| w.len() >= 2)
|
||||
.filter(|w| !STOP_WORDS.contains(w))
|
||||
.filter(|w| !w.chars().all(|c| c.is_numeric()))
|
||||
.filter(|w| seen.insert(*w))
|
||||
.map(|w| w.to_string())
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_removes_stop_words() {
|
||||
let kw = extract_keywords("that thing we discussed about the API");
|
||||
assert_eq!(kw, vec!["discussed", "api"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_all_stop_words_returns_empty() {
|
||||
let kw = extract_keywords("what is that?");
|
||||
assert!(kw.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_preserves_meaningful_words() {
|
||||
let kw = extract_keywords("rust programming async patterns");
|
||||
assert_eq!(kw, vec!["rust", "programming", "async", "patterns"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_filters_single_char_words() {
|
||||
let kw = extract_keywords("I a x language");
|
||||
// "i" = 1 char, "a" = 1 char, "x" = 1 char → all filtered by length
|
||||
assert_eq!(kw, vec!["language"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_preserves_short_meaningful_terms() {
|
||||
// 2-char terms that are meaningful in programming should survive
|
||||
let kw = extract_keywords("Go and JS patterns");
|
||||
assert_eq!(kw, vec!["go", "js", "patterns"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_short_stop_words_filtered() {
|
||||
// 2-char stop words ("is", "it", "do", "we") should still be removed
|
||||
let kw = extract_keywords("is it ok to do that");
|
||||
assert_eq!(kw, vec!["ok"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_filters_pure_numbers() {
|
||||
let kw = extract_keywords("port 8080 and 443 config");
|
||||
assert_eq!(kw, vec!["port", "config"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_deduplicates() {
|
||||
let kw = extract_keywords("rust rust rust programming");
|
||||
assert_eq!(kw, vec!["rust", "programming"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_handles_punctuation() {
|
||||
let kw = extract_keywords("what's the solution for the bug?");
|
||||
assert_eq!(kw, vec!["solution", "bug"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_preserves_underscored_identifiers() {
|
||||
let kw = extract_keywords("the my_function variable");
|
||||
assert_eq!(kw, vec!["my_function", "variable"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_empty_query() {
|
||||
assert!(extract_keywords("").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_only_punctuation() {
|
||||
assert!(extract_keywords("??? !!! ...").is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mixed_case() {
|
||||
let kw = extract_keywords("Rust Programming ASYNC");
|
||||
assert_eq!(kw, vec!["rust", "programming", "async"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_real_conversational_queries() {
|
||||
assert_eq!(
|
||||
extract_keywords("what was the solution for the authentication bug"),
|
||||
vec!["solution", "authentication", "bug"]
|
||||
);
|
||||
assert_eq!(
|
||||
extract_keywords("how do I configure the memory system"),
|
||||
vec!["configure", "memory", "system"]
|
||||
);
|
||||
assert_eq!(
|
||||
extract_keywords("show me that database migration we talked about"),
|
||||
vec!["database", "migration", "talked"]
|
||||
);
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user