用 Java 编写一个轻量级搜索引擎爬虫后端服务

日常开发，经常需要获取搜索引擎的搜索结果，但直接爬取搜索引擎会面临反爬、性能、可用性等一系列问题。本文将分享如何使用 Java 原生 API + 极简第三方库，构建一个带有本地缓存、并发控制与反爬对抗能力的轻量级 HTTP 代理网关（BFF 层），为前端或脚本提供稳定、高质量的搜索数据接口。

---

一、为什么不用 Spring Boot？

Spring Boot 是 Java 生态的标杆，但在这个场景下它过于重了：

• 启动时间数秒，内存占用上百 MB
• 大量用不到的自动配置
• 部署在资源受限的服务器（如树莓派、轻量容器）上并不划算

取而代之，这里采用 JDK 原生 API + 极简第三方库 的组合：

层次	技术选型	理由
HTTP 服务	com.sun.net.httpserver.HttpServer	JDK 自带，毫秒级启动
HTTP 客户端	java.net.http.HttpClient	JDK 11+ 原生，支持 HTTP/2
HTML 解析	Jsoup 1.23.1	最轻量的 DOM 解析器
JSON 序列化	Gson 2.10.1	久经考验，API 简洁

这种组合最终打包的 JAR 文件仅有 800 KB 左右 （含依赖），内存仅需不到 10 MB。

---

二、整体架构设计

整个服务可以看作一个 “带缓存的智能代理网关”，其核心职责是：

1. 接收前端请求（支持 GET/POST，跨域）
2. 校验参数，防止恶意攻击
3. 检查本地缓存，避免重复抓取
4. 必要时从百度抓取，并解析、重排结果
5. 返回结构化 JSON，附带日志追踪

---

三、核心模块

3.1 缓存与并发控制

这是整个服务最复杂的地方。这里使用 ConcurrentHashMap 作为缓存存储，并引入 刷新锁 来防止缓存击穿。

主要设计：

• 缓存有效期：30 分钟（可统一配置）
• 刷新锁：使用 ConcurrentHashMap.putIfAbsent 实现 JVM 级别的互斥，确保同一关键词+页码只有 1 个线程去抓取百度
• SWR 降级（Stale-While-Revalidate）：如果抓取失败但存在过期缓存，则延长缓存 TTL 并返回旧数据，避免服务完全不可用

为什么这样设计？

假如，缓存过期瞬间，有 100 个请求同时涌入。若不加锁，100 个线程会同时抓取搜索引擎，立即触发反爬验证。加锁后只有 1 个线程去抓取，其余 99 个直接返回旧缓存。

3.2 伪装成真实浏览器对抗反爬

搜索引擎都拥有强大的反爬系统，单纯修改 User-Agent 完全不够。程序需要在 HTTP 协议层 深度伪装：

• 请求头顺序与内容：完全模拟 Chrome/Edge 的典型请求头，包括 Accept、Accept-Language、DNT、UA-CPU 等非标准头
• Referer 策略：强制设置为 https://www.baidu.com/，模拟从首页点击搜索
• Cookie 预热：正式搜索前先请求百度首页，获取必要的 Cookie（如 BAIDUID）
• 手动处理压缩：虽然 HttpClient 支持自动解压，但手动检查 Content-Encoding 并调用 GZIPInputStream/InflaterInputStream 解压，可应对各种不规范网关

核心代码：

// 构建请求时添加伪装头
HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create(url))
    .header("Accept", "text/html, application/xhtml+xml, image/jxr, ...")
    .header("User-Agent", "Mozilla/5.0 ... Edg/109.0.1518.140")
    .header("UA-CPU", "AMD64")
    .header("Referer", "https://www.baidu.com/")
    .timeout(Duration.ofSeconds(15))
    .build();

3.3 结果重排

很多搜索引擎的原生结果中常混有 SEO 站点、广告和低质量内容。因此，本程序，维护了一个 高权重域名白名单，如：

baike.baidu.com, zhihu.com, douban.com

提取结果后，根据 URL 的域名匹配白名单，将匹配到的结果置顶，从而保证前端获得更权威、更有用的信息。

3.4 规范化日志

在没有 APM 系统的单体服务中，日志是排查问题的唯一手段。本程序设计了统一的日志格式：

[时间] [级别] [RequestID] [客户端IP] 方法 路径 - 类型 - 状态码 - 耗时 - 关键词 页码 - 消息

• RequestID 使用 AtomicLong 自增，贯穿请求全链路
• 类型包含 CACHE_HIT、CACHE_REFRESH、CACHE_EXTEND、ERROR 等，清晰描述处理路径
• 耗时统计帮助定位性能瓶颈

---

四、完整代码

WebSearch.java：

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

import java.net.URI;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.net.InetSocketAddress;
import java.net.BindException;
import java.net.ConnectException;
import java.net.SocketTimeoutException;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.CookieManager;
import java.net.CookiePolicy;
import java.time.Duration;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
import java.util.zip.GZIPInputStream;
import java.util.zip.InflaterInputStream;

import com.sun.net.httpserver.HttpServer;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;

public class WebSearch {
    private static final HttpClient client = HttpClient.newBuilder()
            .version(HttpClient.Version.HTTP_1_1)
            .followRedirects(HttpClient.Redirect.NORMAL)
            .connectTimeout(Duration.ofSeconds(10))
            .cookieHandler(new CookieManager(null, CookiePolicy.ACCEPT_ALL))
            .build();

    // Priority domain list (ordered)
    private static final List<String> PRIORITY_DOMAINS = Arrays.asList(
            "baike.baidu.com",
            "zhihu.com",
            "douban.com",
            "gov.cn",
            "edu.cn",
            "baijiahao.baidu.com",
            "jianshu.com",
            "sohu.com",
            "163.com",
            "people.cn",
            "news.cn",
            "qq.com"
    );

    // Error codes
    private static final int SUCCESS = 0;
    private static final int ERR_MISSING_KW = 1001;
    private static final int ERR_INVALID_PG = 1002;
    private static final int ERR_SECURITY_VERIFICATION = 1003;
    private static final int ERR_NETWORK = 1004;
    private static final int ERR_PARSE = 1005;
    private static final int ERR_METHOD_NOT_ALLOWED = 1006;
    private static final int ERR_UNKNOWN = 1099;

    // Gson instance
    private static final Gson gson = new GsonBuilder().setPrettyPrinting().create();

    // Cache TTL: 30 minutes
    private static final long CACHE_TTL_MS = 30 * 60 * 1000;
    // Cache cleaner interval: 5 minutes
    private static final long CLEANER_INTERVAL_MINUTES = 5;

    // Cache storage
    private static final ConcurrentHashMap<String, CacheEntry> cache = new ConcurrentHashMap<>();
    // Refresh lock: prevents concurrent refresh for same key
    private static final ConcurrentHashMap<String, Boolean> refreshing = new ConcurrentHashMap<>();

    // Request ID generator
    private static final AtomicLong requestCounter = new AtomicLong(0);

    // Cache entry
    static class CacheEntry {
        final Map<String, Object> data;
        volatile long expireTime; // absolute timestamp

        CacheEntry(Map<String, Object> data, long expireTime) {
            this.data = data;
            this.expireTime = expireTime;
        }

        boolean isExpired() {
            return System.currentTimeMillis() > expireTime;
        }
    }

    // Cache cleaner thread pool
    private static final ScheduledExecutorService cleaner = Executors.newSingleThreadScheduledExecutor();

    // Start the cache cleaner
    private static void startCacheCleaner() {
        cleaner.scheduleAtFixedRate(() -> {
            try {
                int removed = 0;
                for (Map.Entry<String, CacheEntry> entry : cache.entrySet()) {
                    String key = entry.getKey();
                    CacheEntry val = entry.getValue();
                    if (val.isExpired() && !refreshing.containsKey(key)) {
                        cache.remove(key);
                        removed++;
                    }
                }
                if (removed > 0) {
                    String timestamp = java.time.LocalDateTime.now()
                            .format(java.time.format.DateTimeFormatter.ISO_LOCAL_DATE_TIME);
                    System.out.printf("[%s] [INFO] [CLEANER] CACHE_CLEAN - Removed %d expired entries%n",
                            timestamp, removed);
                }
            } catch (Exception e) {
                String timestamp = java.time.LocalDateTime.now()
                        .format(java.time.format.DateTimeFormatter.ISO_LOCAL_DATE_TIME);
                System.err.printf("[%s] [ERROR] [CLEANER] CACHE_CLEAN - Cleaner error: %s%n",
                        timestamp, e.getMessage());
            }
        }, CLEANER_INTERVAL_MINUTES, CLEANER_INTERVAL_MINUTES, TimeUnit.MINUTES);
    }

    private static int getDomainPriority(String url) {
        try {
            String host = URI.create(url).getHost();
            if (host == null) return Integer.MAX_VALUE;
            host = host.toLowerCase();
            if (host.startsWith("www.")) host = host.substring(4);
            for (int i = 0; i < PRIORITY_DOMAINS.size(); i++) {
                String domain = PRIORITY_DOMAINS.get(i).toLowerCase();
                if (host.equals(domain) || host.endsWith("." + domain)) {
                    return i;
                }
            }
        } catch (Exception e) { /* ignore */ }
        return Integer.MAX_VALUE;
    }

    private static String fetch(String url) throws Exception {
        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(url))
                .header("Accept", "text/html, application/xhtml+xml, image/jxr, application/octet-stream, */*")
                .header("Accept-Encoding", "gzip, deflate")
                .header("Accept-Language", "*")
                .header("dnt", "1")
                .header("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win32; X86) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Edg/109.0.1518.140")
                .header("UA-CPU", "AMD64")
                .header("Referer", "https://www.baidu.com/")
                .timeout(Duration.ofSeconds(15))
                .build();

        try {
            HttpResponse<byte[]> response = client.send(request, HttpResponse.BodyHandlers.ofByteArray());
            byte[] body = response.body();

            String contentEncoding = response.headers().firstValue("Content-Encoding").orElse("");
            if (contentEncoding.contains("gzip")) {
                try (GZIPInputStream gzip = new GZIPInputStream(new ByteArrayInputStream(body));
                     ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
                    byte[] buffer = new byte[8192];
                    int len;
                    while ((len = gzip.read(buffer)) > 0) baos.write(buffer, 0, len);
                    body = baos.toByteArray();
                }
            } else if (contentEncoding.contains("deflate")) {
                try (InflaterInputStream inflater = new InflaterInputStream(new ByteArrayInputStream(body));
                     ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
                    byte[] buffer = new byte[8192];
                    int len;
                    while ((len = inflater.read(buffer)) > 0) baos.write(buffer, 0, len);
                    body = baos.toByteArray();
                }
            }
            return new String(body, StandardCharsets.UTF_8);
        } catch (ConnectException | SocketTimeoutException e) {
            throw new Exception("Network connection error: " + e.getMessage(), e);
        }
    }

    private static Map<String, Object> parseWithJsoup(String html, String keyword, int currentPage) {
        Document doc = Jsoup.parse(html);
        Elements resultTables = doc.select("table.result");

        List<Map<String, String>> results = new ArrayList<>();
        for (Element table : resultTables) {
            Element titleLink = table.select("h3.t a").first();
            if (titleLink == null) continue;
            String title = titleLink.text();
            String url = titleLink.attr("href");
            Element font = table.select("font[size=-1]").first();
            String description = (font != null) ? font.text() : "";
            description = description.replaceAll("\\s*-\\s*百度快照\\s*$", "");

            Map<String, String> item = new HashMap<>();
            item.put("title", title);
            item.put("url", url);
            item.put("description", description);
            results.add(item);
        }

        results.sort(Comparator.comparingInt((Map<String, String> item) -> getDomainPriority(item.get("url"))));

        Elements navLinks = doc.select("a.n");
        boolean hasPreviousPage = false, hasNextPage = false;
        for (Element a : navLinks) {
            String text = a.text();
            if (text.contains("上一页") || text.contains("<上一页")) hasPreviousPage = true;
            if (text.contains("下一页") || text.contains("下一页>")) hasNextPage = true;
        }

        Map<String, Object> resultMap = new HashMap<>();
        resultMap.put("keyword", keyword);
        resultMap.put("currentPage", currentPage);
        resultMap.put("hasPreviousPage", hasPreviousPage);
        resultMap.put("hasNextPage", hasNextPage);
        resultMap.put("results", results);
        return resultMap;
    }

    // ----- HTTP Handler -----
    static class SearchHandler implements HttpHandler {
        @Override
        public void handle(HttpExchange exchange) throws IOException {
            long startTime = System.currentTimeMillis();
            String requestId = String.valueOf(requestCounter.incrementAndGet());
            String clientIp = exchange.getRemoteAddress().toString();
            String method = exchange.getRequestMethod();
            String path = exchange.getRequestURI().toString();

            // Handle OPTIONS preflight
            if ("OPTIONS".equalsIgnoreCase(method)) {
                setCorsHeaders(exchange);
                exchange.sendResponseHeaders(200, -1);
                // Log OPTIONS request (optional)
                String timestamp = java.time.LocalDateTime.now()
                        .format(java.time.format.DateTimeFormatter.ISO_LOCAL_DATE_TIME);
                System.out.printf("[%s] [INFO] [%s] [%s] %s %s - OPTIONS - 200 - 0ms - - - Preflight handled%n",
                        timestamp, requestId, clientIp, method, path);
                return;
            }

            // Allow only GET and POST
            if (!"GET".equalsIgnoreCase(method) && !"POST".equalsIgnoreCase(method)) {
                sendJsonError(exchange, 405, ERR_METHOD_NOT_ALLOWED, "Only GET and POST methods are allowed", null);
                long duration = System.currentTimeMillis() - startTime;
                log(requestId, clientIp, method, path, "ERROR", 405, duration, null, -1,
                        "Method not allowed: " + method);
                return;
            }

            // Parse parameters
            String query = exchange.getRequestURI().getRawQuery();
            if ("POST".equalsIgnoreCase(method)) {
                String body = new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8);
                if (query == null) query = body;
                else query += "&" + body;
            }

            Map<String, String> params = parseQuery(query);
            String kw = params.get("kw");
            String pgStr = params.get("pg");

            // Validate kw
            if (kw == null || kw.trim().isEmpty()) {
                sendJsonError(exchange, 400, ERR_MISSING_KW, "Missing required parameter: kw (search keyword)", null);
                long duration = System.currentTimeMillis() - startTime;
                log(requestId, clientIp, method, path, "ERROR", 400, duration, null, -1,
                        "Missing keyword parameter");
                return;
            }
            kw = kw.trim();

            // Limit keyword length to 200 chars
            if (kw.length() > 200) {
                sendJsonError(exchange, 400, ERR_INVALID_PG, "Keyword length exceeds 200 characters", null);
                long duration = System.currentTimeMillis() - startTime;
                log(requestId, clientIp, method, path, "ERROR", 400, duration, kw, -1,
                        "Keyword too long: " + kw.length());
                return;
            }

            int page = 0;
            if (pgStr != null && !pgStr.isEmpty()) {
                try {
                    page = Integer.parseInt(pgStr);
                    if (page < 0) {
                        sendJsonError(exchange, 400, ERR_INVALID_PG, "Parameter pg must be a non-negative integer", null);
                        long duration = System.currentTimeMillis() - startTime;
                        log(requestId, clientIp, method, path, "ERROR", 400, duration, kw, page,
                                "Page number negative: " + page);
                        return;
                    }
                    if (page > 100) {
                        sendJsonError(exchange, 400, ERR_INVALID_PG, "Page number cannot exceed 100", null);
                        long duration = System.currentTimeMillis() - startTime;
                        log(requestId, clientIp, method, path, "ERROR", 400, duration, kw, page,
                                "Page number exceeds 100: " + page);
                        return;
                    }
                } catch (NumberFormatException e) {
                    sendJsonError(exchange, 400, ERR_INVALID_PG, "Parameter pg must be an integer (e.g., 0, 1, 2)", null);
                    long duration = System.currentTimeMillis() - startTime;
                    log(requestId, clientIp, method, path, "ERROR", 400, duration, kw, page,
                            "Invalid page format: " + pgStr);
                    return;
                }
            }

            // Build cache key
            String cacheKey = kw + "|" + page;

            // ---- Cache logic ----
            CacheEntry entry = cache.get(cacheKey);
            boolean exists = (entry != null);
            boolean expired = exists && entry.isExpired();

            // Case 1: Cache exists and not expired → return immediately
            if (exists && !expired) {
                sendSuccessResponse(exchange, entry.data, true);
                long duration = System.currentTimeMillis() - startTime;
                log(requestId, clientIp, method, path, "CACHE_HIT", 200, duration, kw, page,
                        "Return cached result");
                return;
            }

            // Case 2: Cache missing or expired → try to acquire refresh lock
            Boolean isRefreshing = refreshing.putIfAbsent(cacheKey, Boolean.TRUE);
            if (isRefreshing == null) {
                // This thread is responsible for refreshing
                log(requestId, clientIp, method, path, "CACHE_REFRESH_START", 0, System.currentTimeMillis() - startTime,
                        kw, page, "Starting refresh");
                Map<String, Object> newData = null;
                Exception error = null;
                try {
                    // Perform search
                    fetch("https://www.baidu.com/");
                    String encodedKeyword = URLEncoder.encode(kw, StandardCharsets.UTF_8.name());
                    int pn = page * 50;
                    String searchUrl = "https://www.baidu.com/s?wd=" + encodedKeyword
                            + "&pn=" + pn
                            + "&tn=baidurt"
                            + "&rn=50"
                            + "&ie=utf-8"
                            + "&oe=utf-8";
                    String html = fetch(searchUrl);

                    if (html.contains("<div class=\"timeout hide-callback\">")) {
                        throw new Exception("Security verification triggered");
                    }

                    newData = parseWithJsoup(html, kw, page);
                    // Success: update cache
                    cache.put(cacheKey, new CacheEntry(newData, System.currentTimeMillis() + CACHE_TTL_MS));
                    sendSuccessResponse(exchange, newData, false);
                    long duration = System.currentTimeMillis() - startTime;
                    log(requestId, clientIp, method, path, "CACHE_REFRESH", 200, duration, kw, page,
                            "Cache updated");
                } catch (Exception e) {
                    error = e;
                    // If we have old cache, extend its TTL and return it
                    if (entry != null) {
                        entry.expireTime = System.currentTimeMillis() + CACHE_TTL_MS;
                        sendSuccessResponse(exchange, entry.data, true);
                        long duration = System.currentTimeMillis() - startTime;
                        log(requestId, clientIp, method, path, "CACHE_EXTEND", 200, duration, kw, page,
                                "Refresh failed, returned stale cache: " + e.getMessage());
                    } else {
                        // No cache at all -> return error
                        int errCode = ERR_UNKNOWN;
                        String msg = "Internal server error";
                        if (e.getMessage() != null && e.getMessage().contains("Network connection error")) {
                            errCode = ERR_NETWORK;
                            msg = "Network connection error, please check network or try later";
                        } else if (e.getCause() instanceof ConnectException || e.getCause() instanceof SocketTimeoutException) {
                            errCode = ERR_NETWORK;
                            msg = "Connection timeout, please check network or try later";
                        } else if (e.getMessage() != null && e.getMessage().toLowerCase().contains("timeout")) {
                            errCode = ERR_NETWORK;
                            msg = "Request timeout";
                        } else if (e.getMessage() != null && e.getMessage().contains("Security verification")) {
                            errCode = ERR_SECURITY_VERIFICATION;
                            msg = "Security verification triggered, please try later or use a different IP";
                        }
                        sendJsonError(exchange, 500, errCode, msg, null);
                        long duration = System.currentTimeMillis() - startTime;
                        log(requestId, clientIp, method, path, "ERROR", 500, duration, kw, page,
                                "Search failed and no cache: " + e.getMessage());
                        // Print stack trace to stderr for debugging
                        e.printStackTrace(System.err);
                    }
                } finally {
                    refreshing.remove(cacheKey); // Release refresh lock
                }
            } else {
                // Another thread is refreshing; return old cache if exists, else error
                if (entry != null) {
                    sendSuccessResponse(exchange, entry.data, true);
                    long duration = System.currentTimeMillis() - startTime;
                    log(requestId, clientIp, method, path, "CACHE_STALE", 200, duration, kw, page,
                            "Return stale while refreshing");
                } else {
                    // No cache and refresh in progress; should rarely happen.
                    sendJsonError(exchange, 503, ERR_UNKNOWN, "Cache is being refreshed, please try again shortly", null);
                    long duration = System.currentTimeMillis() - startTime;
                    log(requestId, clientIp, method, path, "ERROR", 503, duration, kw, page,
                            "No cache and refresh in progress");
                }
            }
        }

        private void log(String requestId, String clientIp, String method, String path,
                         String type, int statusCode, long duration, String keyword, int page,
                         String message) {
            String timestamp = java.time.LocalDateTime.now()
                    .format(java.time.format.DateTimeFormatter.ISO_LOCAL_DATE_TIME);
            String level;
            if (type.startsWith("ERROR")) {
                level = "ERROR";
                System.err.printf("[%s] [%s] [%s] [%s] %s %s - %s - %d - %dms - %s %d - %s%n",
                        timestamp, level, requestId, clientIp, method, path,
                        type, statusCode, duration,
                        (keyword != null ? keyword : "-"),
                        page,
                        message);
            } else if (type.contains("EXTEND") || type.contains("STALE") || type.contains("REFRESH_FAIL")) {
                level = "WARN";
                System.err.printf("[%s] [%s] [%s] [%s] %s %s - %s - %d - %dms - %s %d - %s%n",
                        timestamp, level, requestId, clientIp, method, path,
                        type, statusCode, duration,
                        (keyword != null ? keyword : "-"),
                        page,
                        message);
            } else {
                level = "INFO";
                System.out.printf("[%s] [%s] [%s] [%s] %s %s - %s - %d - %dms - %s %d - %s%n",
                        timestamp, level, requestId, clientIp, method, path,
                        type, statusCode, duration,
                        (keyword != null ? keyword : "-"),
                        page,
                        message);
            }
        }

        private void sendSuccessResponse(HttpExchange exchange, Map<String, Object> data, boolean fromCache) throws IOException {
            Map<String, Object> response = new HashMap<>();
            response.put("code", SUCCESS);
            response.put("message", fromCache ? "cached data" : "success");
            response.put("data", data);
            sendJson(exchange, 200, response);
        }

        private Map<String, String> parseQuery(String query) {
            Map<String, String> params = new HashMap<>();
            if (query == null || query.isEmpty()) return params;
            for (String pair : query.split("&")) {
                int idx = pair.indexOf('=');
                if (idx > 0) {
                    try {
                        String key = URLDecoder.decode(pair.substring(0, idx), StandardCharsets.UTF_8);
                        String val = URLDecoder.decode(pair.substring(idx + 1), StandardCharsets.UTF_8);
                        params.put(key, val);
                    } catch (IllegalArgumentException e) {
                        // ignore malformed key=value pairs
                    }
                }
            }
            return params;
        }

        private void sendJson(HttpExchange exchange, int httpCode, Object obj) throws IOException {
            String json = gson.toJson(obj);
            setCorsHeaders(exchange);
            exchange.getResponseHeaders().set("Content-Type", "application/json; charset=UTF-8");
            byte[] jsonBytes = json.getBytes(StandardCharsets.UTF_8);
            exchange.sendResponseHeaders(httpCode, jsonBytes.length);
            try (OutputStream os = exchange.getResponseBody()) {
                os.write(jsonBytes);
            }
        }

        private void sendJsonError(HttpExchange exchange, int httpCode, int errCode, String message, Map<String, Object> data) throws IOException {
            Map<String, Object> resp = new HashMap<>();
            resp.put("code", errCode);
            resp.put("message", message);
            resp.put("data", data);
            sendJson(exchange, httpCode, resp);
        }

        private void setCorsHeaders(HttpExchange exchange) {
            exchange.getResponseHeaders().set("Access-Control-Allow-Origin", "*");
            exchange.getResponseHeaders().set("Access-Control-Allow-Methods", "GET, POST, OPTIONS");
            exchange.getResponseHeaders().set("Access-Control-Allow-Headers", "Content-Type");
            exchange.getResponseHeaders().set("Access-Control-Max-Age", "86400");
        }
    }

    // ----- Help message -----
    private static void printHelp() {
        System.err.println("WebSearch HTTP Service - Search API Backend");
        System.err.println();
        System.err.println("Usage:");
        System.err.println("  java -jar WebSearch.jar [--host <address>] [--port <port>]");
        System.err.println();
        System.err.println("Options:");
        System.err.println("  --host <address>   Listening address, default 127.0.0.1");
        System.err.println("  --port <port>      Listening port, default 6080");
        System.err.println("  -h, --help         Show this help message");
        System.err.println();
        System.err.println("API Endpoint:");
        System.err.println("  GET/POST /search?kw=<keyword>&pg=<page>");
        System.err.println("    kw : required, search keyword");
        System.err.println("    pg : optional, page number starting from 0, default 0, max 100");
        System.err.println();
        System.err.println("Response (JSON):");
        System.err.println("  {");
        System.err.println("    \"code\": 0,          // 0 success, non-zero error");
        System.err.println("    \"message\": \"...\",  // status message");
        System.err.println("    \"data\": {           // search result on success, null on error");
        System.err.println("      \"keyword\": \"...\",");
        System.err.println("      \"currentPage\": 0,");
        System.err.println("      \"hasPreviousPage\": true/false,");
        System.err.println("      \"hasNextPage\": true/false,");
        System.err.println("      \"results\": [");
        System.err.println("        { \"title\": \"...\", \"url\": \"...\", \"description\": \"...\" }");
        System.err.println("      ]");
        System.err.println("    }");
        System.err.println("  }");
        System.err.println();
        System.err.println("Examples:");
        System.err.println("  curl \"http://127.0.0.1:6080/search?kw=hello&pg=0\"");
        System.err.println("  curl -X POST -d \"kw=hello&pg=1\" http://127.0.0.1:6080/search");
        System.err.println();
        System.err.println("Startup example:");
        System.err.println("  java -jar WebSearch.jar --host 0.0.0.0 --port 8080");
    }

    // ----- Main entry -----
    public static void main(String[] args) {
        // Force UTF-8 for stdout/stderr
        try {
            System.setOut(new PrintStream(System.out, true, StandardCharsets.UTF_8.name()));
            System.setErr(new PrintStream(System.err, true, StandardCharsets.UTF_8.name()));
        } catch (Exception e) { /* ignore */ }

        String host = "127.0.0.1";
        int port = 6080;
        boolean showHelp = false;

        for (int i = 0; i < args.length; i++) {
            String arg = args[i];
            if ("-h".equals(arg) || "--help".equals(arg)) {
                showHelp = true;
                break;
            } else if ("--host".equals(arg)) {
                if (i + 1 < args.length) {
                    host = args[++i];
                } else {
                    System.err.println("Error: --host requires an address");
                    System.exit(1);
                }
            } else if ("--port".equals(arg)) {
                if (i + 1 < args.length) {
                    try {
                        port = Integer.parseInt(args[++i]);
                        if (port < 1 || port > 65535) {
                            System.err.println("Error: port must be between 1 and 65535");
                            System.exit(1);
                        }
                    } catch (NumberFormatException e) {
                        System.err.println("Error: port must be an integer");
                        System.exit(1);
                    }
                } else {
                    System.err.println("Error: --port requires a port number");
                    System.exit(1);
                }
            } else {
                System.err.println("Error: unknown argument " + arg);
                System.err.println("Use -h or --help for usage");
                System.exit(1);
            }
        }

        if (showHelp) {
            printHelp();
            System.exit(0);
        }

        // Start HTTP server
        try {
            HttpServer server = HttpServer.create(new InetSocketAddress(host, port), 0);
            server.createContext("/search", new SearchHandler());
            server.setExecutor(null); // default thread pool
            server.start();
            String timestamp = java.time.LocalDateTime.now()
                    .format(java.time.format.DateTimeFormatter.ISO_LOCAL_DATE_TIME);
            System.out.printf("[%s] [INFO] [MAIN] SERVICE_START - WebSearch HTTP service started at http://%s:%d/search%n",
                    timestamp, host, port);
            System.out.println("Press Ctrl+C to stop");

            // Start cache cleaner
            startCacheCleaner();

            Runtime.getRuntime().addShutdownHook(new Thread(() -> {
                server.stop(0);
                cleaner.shutdown();
                try {
                    if (!cleaner.awaitTermination(5, TimeUnit.SECONDS)) {
                        cleaner.shutdownNow();
                    }
                } catch (InterruptedException ignored) {}
                String stopTime = java.time.LocalDateTime.now()
                        .format(java.time.format.DateTimeFormatter.ISO_LOCAL_DATE_TIME);
                System.out.printf("[%s] [INFO] [MAIN] SERVICE_STOP - Service stopped%n", stopTime);
            }));

            Thread.currentThread().join();
        } catch (BindException e) {
            System.err.printf("Failed to start service: Port %d is already in use. Please change the port or stop the occupying process.%n", port);
            System.exit(1);
        } catch (Exception e) {
            System.err.println("Failed to start service: " + e.getMessage());
            e.printStackTrace(System.err);
            System.exit(1);
        }
    }
}

---

五、外部依赖

程序仅需两个外部 JAR：

<!-- pom.xml 片段（实际使用手动下载） -->
<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.23.1</version>
</dependency>
<dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.10.1</version>
</dependency>

手动下载这两个 JAR 文件，与源码放在同一目录即可编译。

---

六、打包部署

6.1 编译与打包

假设源码文件为 WebSearch.java，依赖 JAR 位于同目录：

# 编译（指定编码和 classpath）
javac -encoding UTF-8 -cp ".;jsoup-1.23.1.jar;gson-2.10.1.jar" WebSearch.java

# 创建临时目录，解压依赖，打包成胖 JAR（包含所有依赖）
mkdir temp && cd temp
jar xf ../jsoup-1.23.1.jar
jar xf ../gson-2.10.1.jar
cp ../*.class .
jar cfe ../WebSearch.jar WebSearch .
cd .. && rm -rf ./temp

生成的 WebSearch.jar 即为可直接运行的胖 JAR。

6.2 部署

1. 确保 Java 已安装（这里以 OpenJDK 17 为例）：

   java -version
   # 输出: openjdk version "17.0.19" ...
   

2. 上传 JAR 到服务器（例如 /opt/websearch/）：

   scp WebSearch.jar user@server:/opt/websearch/
   

3. 创建 systemd 服务：

   sudo nano /etc/systemd/system/websearch.service
   

   写入以下内容：

   [Unit]
   Description=WebSearch HTTP Service
   After=network.target
   
   [Service]
   Type=simple
   User=www-data
   WorkingDirectory=/opt/websearch
   ExecStart=/usr/bin/java -jar /opt/websearch/WebSearch.jar --host 0.0.0.0 --port 6080
   Restart=on-failure
   RestartSec=10
   
   [Install]
   WantedBy=multi-user.target
   

4. 启动服务：

   sudo systemctl daemon-reload
   sudo systemctl enable websearch
   sudo systemctl start websearch
   

5. 验证：

   curl "http://localhost:6080/search?kw=java&pg=0"
   

---

七、运行与监控

启动后，控制台会输出结构化日志，例如：

[2026-08-20T14:32:15.123] [INFO] [0001] [127.0.0.1:54321] GET /search?kw=java&pg=0 - CACHE_HIT - 200 - 12ms - java 0 - Return cached result
[2026-08-20T14:35:00.000] [INFO] [CLEANER] CACHE_CLEAN - Removed 2 expired entries

• 通过 CACHE_HIT 比例可评估缓存效率
• 通过 ERROR 日志可快速定位网络或解析问题
• 定期清理日志可使用 logrotate 管理

---

八、总结

本文主要讲解如何 用 Java 原生 API 实现一个轻量级搜索代理服务，集成缓存、并发控制、反爬伪装、结果重排和结构化日志，在极小的资源占用下提供了高可用的 API 接口。整个项目不依赖任何重型框架，适合部署在资源受限的环境。