http状态码403-403禁止使用Java,但不能使用网络浏览器?

我正在编写一个小型Java程序,以获取给定Google搜索字词的结果数量。 出于某种原因,在Java中我得到了403禁止访问,但在Web浏览器中却得到了正确的结果。 码:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URL;


public class DataGetter {

    public static void main(String[] args) throws IOException {
        getResultAmount("test");
    }

    private static int getResultAmount(String query) throws IOException {
        BufferedReader r = new BufferedReader(new InputStreamReader(new URL("https://www.google.com/search?q=" + query).openConnection()
                .getInputStream()));
        String line;
        String src = "";
        while ((line = r.readLine()) != null) {
            src += line;
        }
        System.out.println(src);
        return 1;
    }

}

错误:

Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://www.google.com/search?q=test
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
    at DataGetter.getResultAmount(DataGetter.java:15)
    at DataGetter.main(DataGetter.java:10)

为什么这样做呢?

Doorknob asked 2020-08-12T09:30:21Z
4个解决方案
94 votes

您只需要设置用户代理标头即可使其工作:

URLConnection connection = new URL("https://www.google.com/search?q=" + query).openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
connection.connect();

BufferedReader r  = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));

StringBuilder sb = new StringBuilder();
String line;
while ((line = r.readLine()) != null) {
    sb.append(line);
}
System.out.println(sb.toString());

从异常堆栈跟踪可以看出,已为您透明地处理了SSL。

但是,获取结果数量并不是真的那么简单,在此之后,您必须通过获取Cookie并解析重定向令牌链接来假冒您是浏览器。

String cookie = connection.getHeaderField( "Set-Cookie").split(";")[0];
Pattern pattern = Pattern.compile("content=\\\"0;url=(.*?)\\\"");
Matcher m = pattern.matcher(response);
if( m.find() ) {
    String url = m.group(1);
    connection = new URL(url).openConnection();
    connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
    connection.setRequestProperty("Cookie", cookie );
    connection.connect();
    r  = new BufferedReader(new InputStreamReader(connection.getInputStream(), Charset.forName("UTF-8")));
    sb = new StringBuilder();
    while ((line = r.readLine()) != null) {
        sb.append(line);
    }
    response = sb.toString();
    pattern = Pattern.compile("<div id=\"resultStats\">About ([0-9,]+) results</div>");
    m = pattern.matcher(response);
    if( m.find() ) {
        long amount = Long.parseLong(m.group(1).replaceAll(",", ""));
        return amount;
    }

}

运行完整的代码,结果得到2930000000L

Esailija answered 2020-08-12T09:30:40Z
2 votes

您可能没有设置正确的标题。 在浏览器中使用LiveHttpHeaders(或同等功能)查看浏览器正在发送的标头,然后在代码中模拟它们。

Kevin Day answered 2020-08-12T09:31:00Z
1 votes

对我来说,它通过添加标题来工作:“接受”:“ * / *”

rpajaziti answered 2020-08-12T09:31:20Z
0 votes

这是因为该站点使用SSL。 尝试使用Jersey HTTP客户端。 您可能还必须了解一些有关HTTPS和证书的信息,但是我认为Jersey可以设置为忽略与实际安全性有关的大多数细节。

answered 2020-08-12T09:31:41Z
translate from https://stackoverflow.com:/questions/13670692/403-forbidden-with-java-but-not-web-browser