摘要：工具和資料群高級爬蟲作者自建群，歡迎加入作者收集的爬蟲相關工具和資料簡介類是以命令行工具為參考，使用標準的實現的工具類。

QQ群 - Javascript高級爬蟲 - 作者自建群，歡迎加入！

awesome-java-crawler - 作者收集的爬蟲相關工具和資料

CUrl類是以命令行工具CUrl為參考，使用標準Java的HttpURLConnection實現的Http工具類。

項目地址: https://github.com/rockswang/...
中央倉庫: https://mvnrepository.com/art...

基于標準Java運行庫的Http類實現，源碼兼容級別為1.6，適用性廣泛，可用于服務端、Android等Java環境

代碼精簡緊湊，僅一個1000余行的Java源文件，無任何外部依賴，可不用Maven直接源碼級重用

完全兼容CUrl命令行工具的常用開關，可直接作為命令行工具替代之

支持所有HTTP Method，支持多文件上傳

通過ThreadLocal解決了標準Java中Cookie只能全局保存的問題，可每線程獨立維護Cookie

可將線程中保存的Cookies序列化保存，方便建立Cookies池

支持HTTP認證，支持HTTPS，可啟用/忽略證書安全

支持每連接代理，支持需認證的HTTP/HTTPS代理

跳轉行為可控制，可獲取到每步跳轉的應答頭信息

支持編程自定義應答解析器

支持失敗重試，可編程自定義可重試異常

例1：POST表單提交

    public void httpPost() {
        CUrl curl = new CUrl("http://httpbin.org/post")
                .data("hello=world&foo=bar")
                .data("foo=overwrite");
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例2：通過Fiddler代理（抓包工具）訪問HTTPS站點

    public void insecureHttpsViaFiddler() {
        CUrl curl = new CUrl("https://httpbin.org/get")
                .proxy("127.0.0.1", 8888) // Use Fiddler to capture & parse HTTPS traffic
                .insecure();  // Ignore certificate check since it"s issued by Fiddler
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例3：上傳多個文件，一個內存文件，一個物理文件

    public void uploadMultipleFiles() {
        CUrl.MemIO inMemFile = new CUrl.MemIO();
        try { inMemFile.getOutputStream().write("text file content blabla...".getBytes()); } catch (Exception ignored) {}
        CUrl curl = new CUrl("http://httpbin.org/post")
                .form("formItem", "value") // a plain form item
                .form("file", inMemFile)           // in-memory "file"
                .form("image", new CUrl.FileIO("D:	mpa2.png")); // A file in storage
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例4：模擬手機瀏覽器上的AJAX請求，添加自定義請求頭

    public void customUserAgentAndHeaders() {
        String mobileUserAgent = "Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; KNT-AL10 Build/HUAWEIKNT-AL10) " 
                + "AppleWebKit/537.36 (KHTML, like Gecko) MQQBrowser/7.3 Chrome/37.0.0.0 Mobile Safari/537.36";
        Map fakeAjaxHeaders = new HashMap();
        fakeAjaxHeaders.put("X-Requested-With", "XMLHttpRequest");
        fakeAjaxHeaders.put("Referer", "http://somesite.com/fake_referer");
        CUrl curl = new CUrl("http://httpbin.org/get")
                .opt("-A", mobileUserAgent) // simulate a mobile browser
                .headers(fakeAjaxHeaders)   // simulate an AJAX request
                .header("X-Auth-Token: xxxxxxx"); // other custom header, this might be calculated elsewhere
        curl.exec();
        assertEquals(200, curl.getHttpCode());
    }

例5：多線程并發請求，線程間Cookies相互獨立

    public void threadSafeCookies() {
        final CountDownLatch count = new CountDownLatch(3);
        final CUrl[] curls = new CUrl[3];
        for (int i = 3; --i >= 0;) {
            final int idx = i;
            new Thread() {
                public void run() {
                    CUrl curl = curls[idx] = new CUrl("http://httpbin.org/get")
                            .cookie("thread" + idx + "=#" + idx);
                    curl.exec();
                    count.countDown();
                }
            }.start();
        }
        try { count.await(); } catch (Exception ignored) {} // make sure all requests are done
        assertEquals(200, curls[0].getHttpCode());
        assertEquals("thread0=#0", deepGet(curls[0].getStdout(jsonResolver, null), "headers.Cookie"));
        assertEquals("thread1=#1", deepGet(curls[1].getStdout(jsonResolver, null), "headers.Cookie"));
        assertEquals("thread2=#2", deepGet(curls[2].getStdout(jsonResolver, null), "headers.Cookie"));
    }

例6：編程自定義應答解析器，使用JSoup解析HTML

    private CUrl.Resolver htmlResolver = new CUrl.Resolver() {
        @SuppressWarnings("unchecked")
        @Override
        public Document resolve(int httpCode, byte[] responseBody) throws Throwable {
            String html = new String(responseBody, "UTF-8");
            return Jsoup.parse(html);
        }
    };

    public void customResolver() {
        CUrl curl = new CUrl("http://httpbin.org/html");
        Document html = curl.exec(htmlResolver, null);
        assertEquals(200, curl.getHttpCode());
        assertEquals("Herman Melville - Moby-Dick", html.select("h1:first-child").text());
    }

例7：作為命令行工具使用，請求內容參考例4

java -jar java-curl-1.2.0.jar https://httpbin.org/get ^
    -x 127.0.0.1:8888 -k ^
    -A "Mozilla/5.0 (Linux; U; Android 8.0.0; zh-cn; KNT-AL10 Build/HUAWEIKNT-AL10) AppleWebKit/537.36 (KHTML, like Gecko) MQQBrowser/7.3 Chrome/37.0.0.0 Mobile Safari/537.36" ^
    -H "Referer: http://somesite.com/fake_referer" ^
    -H "X-Requested-With: XMLHttpRequest" ^
    -H "X-Auth-Token: xxxxxxx"