《花100块做个摸鱼小网站! 》第二篇—后端应用搭建和完成第一个爬虫 点击关注公众号，“技术干货”及时达！??基础链接导航?? 服务器 → ?? 阿里云活动地址 看样例 → ?? 摸鱼小网站地址 学代码 → ?? 源码库地址 一、前言大家好呀，我是summo，前面已经教会大家怎么去阿里云买服务器（链接在这，需要自取），以及怎么搭建JDK、Redis、MySQL这些环境或者数据库。从这篇文章开始就进入正式的编码阶段了，我们从后端开始，先把热搜数据获取到，然后再开始前端部分。 本来我想把后端应用搭建和完成第一个爬虫分为两篇文章写的，但是想到墨迹三周还没看到效果，估计有些同学已经等不及了，所以我把这两篇文章合成一篇了，篇幅比较长，感兴趣的同学可以认真看下。因为后端应用初始化的时候涉及很多配置，有很多命名，建议大家写别急着个性化用自己的名字命名应用和配置，可以先用我的命名方式搭建一个出来，先成功再研究个性化，这样成就感很足，学习动力也强。 二、后端应用搭建这个摸鱼小网站主要使用的SpringBoot框架配合一些中间件实现，开发工具用的是idea社区版，建议不要下载人家的正式版然后破解，一来麻烦的很，二来社区版足够开发使用了，点击这个可以下载idea社区版。 1. maven项目搭建 打开idea后，点击New Project，我们从一个空的maven项目开始，一步步把项目搭建出来。 我们输入name、GroupId、ArtifactId点击确定，如果你是老手可以自己搭建，如果你是新手建议跟我输的一样，因为后面还有很多地方要用的这些名字。 把src目录删掉，我们不需要，只留下pom.xml文件就行了。 接下来我们开始创建子module，创建方式如下图： 以summo-sbmy-start子module为例，输入如下的信息就可以创建一个module出来啦。 按照这个方式创建出如下的module出来。 2. pom.xml配置 这部分我现在不详细说，后面我会单独写一篇文章讲我的依赖和配置。如果刚才你的名字取得跟我一样，直接复制粘贴就完事了，如果名字不一样记得把名字给替换为你的。 （1）summo-sbmypom.xml内容如下： 点击展开代码?xml version="1.0" encoding="UTF-8"?project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" modelVersion4.0.0/modelVersion parent groupIdorg.springframework.boot/groupId artifactIdspring-boot-starter-parent/artifactId version2.7.15/version relativePath/!-- lookup parent from repository -- /parent groupIdcom.summo/groupId artifactIdsummo-sbmy/artifactId packagingpom/packaging version1.0-SNAPSHOT/version modules modulesummo-sbmy-dao/module modulesummo-sbmy-service/module modulesummo-sbmy-web/module modulesummo-sbmy-start/module modulesummo-sbmy-job/module modulesummo-sbmy-common/module /modules properties maven.compiler.source/maven.compiler.source maven.compiler.target/maven.compiler.target /properties dependencyManagement dependencies !-- 自依赖 -- dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-common/artifactId version${project.version}/version /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-service/artifactId version${project.version}/version /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-common/artifactId version${project.version}/version /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-dao/artifactId version${project.version}/version /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-job/artifactId version${project.version}/version /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-start/artifactId version${project.version}/version /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-web/artifactId version${project.version}/version /dependency !-- xxl-job -- dependency groupIdcom.xuxueli/groupId artifactIdxxl-job-core/artifactId version2.2.0/version /dependency !-- MySQL驱动 -- dependency groupIdmysql/groupId artifactIdmysql-connector-java/artifactId version8.0.20/version /dependency !-- mybatis-plus驱动 -- dependency groupIdcom.baomidou/groupId artifactIdmybatis-plus-boot-starter/artifactId version3.3.2/version /dependency dependency groupIdcom.baomidou/groupId artifactIdmybatis-plus-extension/artifactId version3.3.2/version /dependency !-- 分页插件 -- dependency groupIdcom.github.pagehelper/groupId artifactIdpagehelper-spring-boot-starter/artifactId version1.4.1/version /dependency !-- lombok -- dependency groupIdorg.projectlombok/groupId artifactIdlombok/artifactId version1.16.22/version /dependency !-- druid链接池 -- dependency groupIdcom.alibaba/groupId artifactIddruid-spring-boot-starter/artifactId version1.1.22/version /dependency !-- aspectj -- dependency groupIdorg.aspectj/groupId artifactIdaspectjweaver/artifactId version1.9.5/version /dependency !-- fastjson -- dependency groupIdcom.alibaba/groupId artifactIdfastjson/artifactId version2.0.21/version /dependency !-- 线程上下文 -- dependency groupIdcom.alibaba/groupId artifactIdtransmittable-thread-local/artifactId version2.11.1/version /dependency !-- 接口参数校验 -- dependency groupIdjavax.validation/groupId artifactIdvalidation-api/artifactId version2.0.1.Final/version /dependency !-- Redisson分布式锁 -- dependency groupIdorg.redisson/groupId artifactIdredisson-spring-boot-starter/artifactId version3.24.0/version /dependency !-- 通用工具类 -- dependency groupIdorg.apache.commons/groupId artifactIdcommons-lang3/artifactId version3.5/version /dependency dependency groupIdorg.apache.commons/groupId artifactIdcommons-collections4/artifactId version4.1/version /dependency !-- VM 模板 -- dependency groupIdcom.alibaba.boot/groupId artifactIdvelocity-spring-boot-starter/artifactId version1.0.4.RELEASE/version /dependency !-- guava -- dependency groupIdcom.google.guava/groupId artifactIdguava/artifactId version32.1.1-jre/version /dependency !-- httpclient -- dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpcore/artifactId version4.4.16/version /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId version4.5.14/version /dependency !-- jsoup -- dependency groupIdorg.jsoup/groupId artifactIdjsoup/artifactId version1.12.1/version /dependency !-- ip2region -- dependency groupIdorg.lionsoul/groupId artifactIdip2region/artifactId version2.6.3/version /dependency !-- 用于读取ip2region.xdb文件使用 -- dependency groupIdcommons-io/groupId artifactIdcommons-io/artifactId version2.6/version /dependency !-- 加解密代码-- dependency groupIdorg.bouncycastle/groupId artifactIdbcprov-jdk15on/artifactId version1.68/version /dependency dependency groupIdcommons-codec/groupId artifactIdcommons-codec/artifactId version1.15/version /dependency /dependencies /dependencyManagement build plugins plugin groupIdorg.apache.maven.plugins/groupId artifactIdmaven-compiler-plugin/artifactId version3.8.1/version configuration source1.8/source target1.8/target encodingutf-8/encoding /configuration /plugin plugin groupIdorg.apache.maven.plugins/groupId artifactIdmaven-source-plugin/artifactId version2.2/version executions execution attach-sources/id goals goaljar/goal /goals /execution /executions configuration finalName${project.build.finalName}/finalName /configuration /plugin /plugins /build/project（2）summo-sbmy-startpom.xml内容如下： 点击展开代码?xml version="1.0" encoding="UTF-8"?project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" parent artifactIdsummo-sbmy/artifactId groupIdcom.summo/groupId version1.0-SNAPSHOT/version /parent modelVersion4.0.0/modelVersion artifactIdsummo-sbmy-start/artifactId properties maven.compiler.source/maven.compiler.source maven.compiler.target/maven.compiler.target /properties dependencies dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-service/artifactId /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-job/artifactId /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-dao/artifactId /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-common/artifactId /dependency dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-web/artifactId /dependency !-- 测试组件 -- dependency groupIdorg.springframework.boot/groupId artifactIdspring-boot-starter-test/artifactId scopetest/scope /dependency dependency groupIdjunit/groupId artifactIdjunit/artifactId scopetest/scope /dependency /dependencies !-- 添加四个环境的变量，变量名为environment -- profiles profile test/id activation activeByDefaulttrue/activeByDefault /activation properties environmenttest/environment /properties /profile profile consumer/id properties environmentconsumer/environment /properties /profile profile producer/id properties environmentproducer/environment /properties /profile /profiles build finalNamesummo-sbmy/finalName resources resource !-- 指定配置文件所在的resource目录 -- directorysrc/main/resources/directory includes includeapplication.properties/include includelogback-spring.xml/include include**/*.html/include include**/*.js/include include**/*.css/include /includes filteringtrue/filtering /resource resource !-- 指定配置文件所在的resource目录 -- directorysrc/main/resources/directory includes include**/*.woff/include include**/*.ttf/include include**/*.xdb/include include**/*.jks/include /includes filteringfalse/filtering /resource /resources plugins plugin groupIdorg.springframework.boot/groupId artifactIdspring-boot-maven-plugin/artifactId version2.1.13.RELEASE/version executions execution goals goalrepackage/goal /goals /execution /executions configuration mainClasscom.summo.sbmy.Application/mainClass /configuration /plugin !-- 解压fat jar到target/${project-name}目录 -- plugin artifactIdmaven-antrun-plugin/artifactId executions execution phasepackage/phase configuration target unzip src="${project.build.directory}/${project.build.finalName}.${project.packaging}" dest="${project.build.directory}/summo-sbmy"/ /target /configuration goals goalrun/goal /goals /execution /executions /plugin /plugins /build/project （3）summo-sbmy-commonpom.xml内容如下： 点击展开代码?xml version="1.0" encoding="UTF-8"?project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" parent artifactIdsummo-sbmy/artifactId groupIdcom.summo/groupId version1.0-SNAPSHOT/version /parent modelVersion4.0.0/modelVersion artifactIdsummo-sbmy-common/artifactId properties maven.compiler.source/maven.compiler.source maven.compiler.target/maven.compiler.target /properties dependencies !-- SpringBoot框架 -- dependency groupIdorg.springframework.boot/groupId artifactIdspring-boot-starter-web/artifactId /dependency !-- lombok -- dependency groupIdorg.projectlombok/groupId artifactIdlombok/artifactId /dependency !-- aspectj -- dependency groupIdorg.aspectj/groupId artifactIdaspectjweaver/artifactId version1.9.5/version /dependency !-- logback核心组件 -- dependency groupIdch.qos.logback/groupId artifactIdlogback-core/artifactId /dependency !-- fastjson -- dependency groupIdcom.alibaba/groupId artifactIdfastjson/artifactId /dependency !-- 线程上下文 -- dependency groupIdcom.alibaba/groupId artifactIdtransmittable-thread-local/artifactId /dependency !-- 分页插件 -- dependency groupIdcom.github.pagehelper/groupId artifactIdpagehelper-spring-boot-starter/artifactId version1.4.1/version /dependency !-- 接口参数校验 -- dependency groupIdjavax.validation/groupId artifactIdvalidation-api/artifactId /dependency dependency groupIdcom.squareup.okhttp3/groupId artifactIdokhttp/artifactId /dependency !-- Redis框架 -- dependency groupIdorg.springframework.boot/groupId artifactIdspring-boot-starter-data-redis/artifactId /dependency !-- Redisson分布式锁 -- dependency groupIdorg.redisson/groupId artifactIdredisson-spring-boot-starter/artifactId /dependency !-- jedis?分布式锁 -- dependency groupIdredis.clients/groupId artifactIdjedis/artifactId /dependency !-- 通用工具类 -- dependency groupIdorg.apache.commons/groupId artifactIdcommons-lang3/artifactId /dependency dependency groupIdorg.apache.commons/groupId artifactIdcommons-collections4/artifactId /dependency !-- guava -- dependency groupIdcom.google.guava/groupId artifactIdguava/artifactId /dependency !-- httpclient -- dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpcore/artifactId /dependency dependency groupIdorg.apache.httpcomponents/groupId artifactIdhttpclient/artifactId /dependency dependency groupIdorg.jsoup/groupId artifactIdjsoup/artifactId /dependency !-- ip2region -- dependency groupIdorg.lionsoul/groupId artifactIdip2region/artifactId /dependency !-- 用于读取ip2region.xdb文件使用 -- dependency groupIdcommons-io/groupId artifactIdcommons-io/artifactId /dependency !-- 加解密代码 -- dependency groupIdorg.bouncycastle/groupId artifactIdbcprov-jdk15on/artifactId /dependency dependency groupIdcommons-codec/groupId artifactIdcommons-codec/artifactId /dependency dependency groupIdcom.google.code.gson/groupId artifactIdgson/artifactId version2.8.6/version /dependency /dependencies/project （4）summo-sbmy-daopom.xml内容如下： 点击展开代码?xml version="1.0" encoding="UTF-8"?project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" parent artifactIdsummo-sbmy/artifactId groupIdcom.summo/groupId version1.0-SNAPSHOT/version /parent modelVersion4.0.0/modelVersion artifactIdsummo-sbmy-dao/artifactId properties maven.compiler.source/maven.compiler.source maven.compiler.target/maven.compiler.target /properties dependencies dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-common/artifactId /dependency !-- MySQL驱动 -- dependency groupIdmysql/groupId artifactIdmysql-connector-java/artifactId /dependency !-- mybatis-plus驱动 -- dependency groupIdcom.baomidou/groupId artifactIdmybatis-plus-boot-starter/artifactId /dependency dependency groupIdcom.baomidou/groupId artifactIdmybatis-plus-extension/artifactId /dependency dependency groupIdjavax.persistence/groupId artifactIdjavax.persistence-api/artifactId /dependency !-- druid链接池 -- dependency groupIdcom.alibaba/groupId artifactIddruid-spring-boot-starter/artifactId /dependency /dependencies build plugins plugin groupIdorg.mybatis.generator/groupId artifactIdmybatis-generator-maven-plugin/artifactId version1.3.6/version configuration configurationFile${basedir}/src/main/resources/generator/generatorConfiguration.xml /configurationFile overwritetrue/overwrite verbosetrue/verbose /configuration dependencies dependency groupIdtk.mybatis/groupId artifactIdmapper/artifactId version4.1.2/version /dependency /dependencies /plugin /plugins /build/project （5）summo-sbmy-servicepom.xml内容如下： 点击展开代码?xml version="1.0" encoding="UTF-8"?project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" parent artifactIdsummo-sbmy/artifactId groupIdcom.summo/groupId version1.0-SNAPSHOT/version /parent modelVersion4.0.0/modelVersion artifactIdsummo-sbmy-service/artifactId properties maven.compiler.source/maven.compiler.source maven.compiler.target/maven.compiler.target /properties dependencies dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-dao/artifactId /dependency /dependencies/project （6）summo-sbmy-webpom.xml内容如下： 点击展开代码?xml version="1.0" encoding="UTF-8"?project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" parent artifactIdsummo-sbmy/artifactId groupIdcom.summo/groupId version1.0-SNAPSHOT/version /parent modelVersion4.0.0/modelVersion artifactIdsummo-sbmy-web/artifactId properties maven.compiler.source/maven.compiler.source maven.compiler.target/maven.compiler.target /properties dependencies dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-service/artifactId /dependency !-- thymeleaf -- dependency groupIdorg.springframework.boot/groupId artifactIdspring-boot-starter-thymeleaf/artifactId /dependency /dependencies/project（7）summo-sbmy-jobpom.xml内容如下： 点击展开代码?xml version="1.0" encoding="UTF-8"?project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" parent artifactIdsummo-sbmy/artifactId groupIdcom.summo/groupId version1.0-SNAPSHOT/version /parent modelVersion4.0.0/modelVersion artifactIdsummo-sbmy-job/artifactId properties maven.compiler.source/maven.compiler.source maven.compiler.target/maven.compiler.target /properties dependencies dependency groupIdcom.summo/groupId artifactIdsummo-sbmy-service/artifactId /dependency /dependencies/projectpom.xml配置贴好后，更新一下，把依赖包都下载下来。如果你发现下载的非常慢，把maven仓库的镜像换一下，换成阿里云的。镜像配置如下： mirrors mirror alimaven/id mirrorOfcentral/mirrorOf namealiyun maven/name urlhttps://maven.aliyun.com/repository/public/url /mirror/mirrors3. application.properties配置 ## 应用名spring.application.name=summo-sbmy## 端口号server.port=8080 # 配置Druid数据源类型spring.datasource.type=com.alibaba.druid.pool.DruidDataSource# 数据库连接URL，包括数据库名、允许的公开密钥检索、字符编码、禁用SSL、时区设置等spring.datasource.url=jdbc:mysql://xxx:3306/summo-sbmy?allowPublicKeyRetrieval=true&characterEncoding=utf8&useSSL=false&serverTimezone=Asia/Shanghai&rewriteBatchedStatements=true&zeroDateTimeBehavior=convertToNull# 数据库用户名spring.datasource.username=xxx# 数据库密码spring.datasource.password=xxx# 数据库驱动类名spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver# 初始化连接池时创建的连接数量spring.datasource.druid.initial-size=5# 连接池最大活跃连接数spring.datasource.druid.max-active=30# 连接池最小空闲连接数spring.datasource.druid.min-idle=5# 等待连接获取的最大等待时间spring.datasource.druid.max-wait=60000# 配置间隔多久才进行一次检测，检测需要关闭的空闲连接spring.datasource.druid.time-between-eviction-runs-millis=60000# 配置一个连接在池中最小生存的时间spring.datasource.druid.min-evictable-idle-time-millis=300000# 验证数据库连接有效的SQL语句spring.datasource.druid.validation-query=SELECT 1 FROM DUAL# 是否在从连接池获取连接前进行检验(建议关闭，影响性能)spring.datasource.druid.test-while-idle=true# 获取连接时执行validationQuery检测连接是否有效(建议关闭，影响性能)spring.datasource.druid.test-on-borrow=false# 归还连接时执行validationQuery检测连接是否有效(建议关闭，影响性能)spring.datasource.druid.test-on-return=false# 是否开启PSCache（PreparedStatement缓存），默认falsespring.datasource.druid.pool-prepared-statements=false# 指定每个连接上PSCache的大小，默认-1表示不限制spring.datasource.druid.max-pool-prepared-statement-per-connection-size=0# 启用监控统计和日志过滤器spring.datasource.druid.filters=stat,wall# 配置StatFilter的参数，合并SQL记录spring.datasource.druid.connection-properties=druid.stat.mergeSql=true;druid.stat.slowSqlMillis=500# 是否合并Druid数据源的监控信息spring.datasource.druid.use-global-data-source-stat=true# 开启Wall过滤器，并指定数据库类型为MySQLspring.datasource.druid.filter.wall.enabled=truespring.datasource.druid.filter.wall.db-type=mysql# 配置StatFilter的数据库类型为MySQLspring.datasource.druid.filter.stat.db-type=mysql# 开启StatFilterspring.datasource.druid.filter.stat.enabled=true # MyBatis配置：自动映射行为设置为全字段mybatis.configuration.auto-mapping-behavior=full# MyBatis配置：下划线转驼峰命名规则mybatis.configuration.map-underscore-to-camel-case=true# MyBatis-Plus的Mapper文件位置mybatis-plus.mapper-locations=classpath*:/mybatis/mapper/*.xml # Redis数据库索引spring.redis.database=0# Redis连接超时时间（毫秒）spring.redis.timeout=1800000# Redis服务器地址spring.redis.host=127.0.0.1# Redis服务器端口spring.redis.port=6379# Redis服务器连接密码spring.redis.password=xxx# 使用Lettuce连接池时的最大等待时间（-1表示无限制）spring.redis.lettuce.pool.max-wait=-1# Lettuce连接池的最大空闲连接数spring.redis.lettuce.pool.max-idle=5# Lettuce连接池的最小空闲连接数spring.redis.lettuce.pool.min-idle=0# Lettuce连接池的最大活跃连接数spring.redis.lettuce.pool.max-active=20# 使用Jedis连接池时的最小空闲连接数spring.redis.jedis.pool.min-idle=8# Jedis连接池的最大空闲连接数spring.redis.jedis.pool.max-idle=500# Jedis连接池的最大活跃连接数spring.redis.jedis.pool.max-active=2000# Jedis连接池的最大等待时间（毫秒）spring.redis.jedis.pool.max-wait=10000 4. logback-spring.xml配置 点击展开代码configuration !-- 默认的一些配置 -- include resource="org/springframework/boot/logging/logback/defaults.xml"/ !-- 定义应用名称，区分应用 -- property name="APP_NAME" value="summo-sbmy"/ !-- 定义日志文件的输出路径 -- property name="LOG_PATH" value="${user.home}/logs/${APP_NAME}"/ !-- 定义日志文件名称和路径 -- property name="LOG_FILE" value="${LOG_PATH}/application.log"/ !-- 定义警告级别日志文件名称和路径 -- property name="WARN_LOG_FILE" value="${LOG_PATH}/warn.log"/ !-- 定义错误级别日志文件名称和路径 -- property name="ERROR_LOG_FILE" value="${LOG_PATH}/error.log"/ !-- 自定义控制台打印格式 -- property name="FILE_LOG_PATTERN" value="%green(%d{yyyy-MM-dd HH:mm:ss.SSS}) [%blue(requestId: %X{requestId})] [%highlight(%thread)] ${PID:- } %logger{36} %-5level - %msg%n"/ !-- 将日志滚动输出到application.log文件中 -- appender name="APPLICATION" class="ch.qos.logback.core.rolling.RollingFileAppender" !-- 输出文件目的地 -- file${LOG_FILE}/file encoder pattern${FILE_LOG_PATTERN}/pattern charsetutf8/charset /encoder !-- 设置 RollingPolicy 属性，用于配置文件大小限制，保留天数、文件名格式 -- rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy" !-- 文件命名格式 -- fileNamePattern${LOG_FILE}.%d{yyyy-MM-dd}.%i.log/fileNamePattern !-- 文件保留最大天数 -- maxHistory/maxHistory !-- 文件大小限制 -- maxFileSize50MB/maxFileSize !-- 文件总大小 -- totalSizeCap500MB/totalSizeCap /rollingPolicy /appender !-- 摘取出WARN级别日志输出到warn.log中 -- appender name="WARN" class="ch.qos.logback.core.rolling.RollingFileAppender" file${WARN_LOG_FILE}/file encoder !-- 使用默认的输出格式打印 -- pattern${CONSOLE_LOG_PATTERN}/pattern charsetutf8/charset /encoder !-- 设置 RollingPolicy 属性，用于配置文件大小限制，保留天数、文件名格式 -- rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy" !-- 文件命名格式 -- fileNamePattern${LOG_PATH}/warn.%d{yyyy-MM-dd}.%i.log/fileNamePattern !-- 文件保留最大天数 -- maxHistory/maxHistory !-- 文件大小限制 -- maxFileSize50MB/maxFileSize !-- 文件总大小 -- totalSizeCap500MB/totalSizeCap /rollingPolicy !-- 日志过滤器，将WARN相关日志过滤出来 -- filter class="ch.qos.logback.classic.filter.ThresholdFilter" levelWARN/level /filter /appender !-- 摘取出ERROR级别日志输出到error.log中 -- appender name="ERROR" class="ch.qos.logback.core.rolling.RollingFileAppender" file${ERROR_LOG_FILE}/file encoder !-- 使用默认的输出格式打印 -- pattern${CONSOLE_LOG_PATTERN}/pattern charsetutf8/charset /encoder !-- 设置 RollingPolicy 属性，用于配置文件大小限制，保留天数、文件名格式 -- rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy" !-- 文件命名格式 -- fileNamePattern${LOG_PATH}/error.%d{yyyy-MM-dd}.%i.log/fileNamePattern !-- 文件保留最大天数 -- maxHistory/maxHistory !-- 文件大小限制 -- maxFileSize50MB/maxFileSize !-- 文件总大小 -- totalSizeCap500MB/totalSizeCap /rollingPolicy !-- 日志过滤器，将ERROR相关日志过滤出来 -- filter class="ch.qos.logback.classic.filter.ThresholdFilter" levelERROR/level /filter /appender !-- 配置控制台输出 -- appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender" encoder pattern${FILE_LOG_PATTERN}/pattern charsetutf8/charset /encoder /appender !-- 配置输出级别 -- root level="INFO" !-- 加入控制台输出 -- appender-ref ref="CONSOLE"/ !-- 加入APPLICATION输出 -- appender-ref ref="APPLICATION"/ !-- 加入WARN日志输出 -- appender-ref ref="WARN"/ !-- 加入ERROR日志输出 -- appender-ref ref="ERROR"/ /root/configuration5. 创建启动类 package com.summo.sbmy; import org.springframework.boot.SpringApplication;import org.springframework.boot.autoconfigure.SpringBootApplication;import org.springframework.scheduling.annotation.EnableScheduling; /** * @author summo * @version Application.java, 1.0.0 * @description 启动核心类 * @date 2024年08月09 */@SpringBootApplication(scanBasePackages = {"com.summo.sbmy"})@EnableSchedulingpublic class Application { public static void main(String[] args) { SpringApplication.run(Application.class, args); } } 点击启动，打印如下就算成功 三、实现抖音热搜爬虫第一次写爬虫代码，我们找一个最简单，也是最安全的例子，抖音热搜。 1. 爬虫方案评估 为什么简单？它就一个接口：https://www.iesdouyin.com/web/api/v2/hotsearch/billboard/word/，免登录也不需要参数，你们直接在浏览器上调用就可以把抖音的热搜数据获取到了； 为什么安全？调用https://www.iesdouyin.com/robots.txt接口，返回如下： User-agent: *Allow: / Sitemap: http://www.iesdouyin.com/sitemap/index.xml这个代表什么呢？ 看到了吗，人家不仅允许你去爬，还提供了网站地图的链接，这也是他们为了提高谷歌、必应、百度等浏览器的收录的一种办法。所以，不用担心爬这些数据会怎么样了。 2. 获取链接的cURL代码 先上一张图，如下： 大概的步骤如下： 输入https://www.iesdouyin.com/web/api/v2/hotsearch/billboard/word/链接，按下enter；打开控制台，选择【全部】找到刚才调用的接口；选中接口，右键打开菜单，选择复制里面的“以cURL格式复制”这一项。复制出来是这样的东西 curl 'https://www.iesdouyin.com/web/api/v2/hotsearch/billboard/word/' \ -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \ -H 'accept-language: zh-CN,zh;q=0.9' \ -H 'cache-control: no-cache' \ -H 'cookie: ttwid=1%7CJ6ehEognyMAob_gD6oZwA40monN8E_sENr3IUZmuk7o%7C1712472728%7C44b0cd0003fb75861789d62e56f014eaea3d198898a0ae9a947bf61d95d8ac1a; __ac_signature=_02B4Z6wo00f01fFoqvgAAIDBFmj97SX8qiXxSK5AABr708; __ac_referer=https://pre-dc-console.alibaba-inc.com/' \ -H 'pragma: no-cache' \ -H 'priority: u=0, i' \ -H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"' \ -H 'sec-ch-ua-mobile: ?0' \ -H 'sec-ch-ua-platform: "macOS"' \ -H 'sec-fetch-dest: document' \ -H 'sec-fetch-mode: navigate' \ -H 'sec-fetch-site: none' \ -H 'sec-fetch-user: ?1' \ -H 'upgrade-insecure-requests: 1' \ -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36'能看懂吗？能看懂最好，看不懂也没有关系，不是给我们看的，给Postman看的。Postman是一个http接口调用工具非常好用，电脑上没有Postman的同学就去下载一个。 3. 使用Postman生成调用代码 打开你的Postman软件，按照我下面这张图操作： 按照我上面的步骤，将cURL命令导入Postman，可以快速生成一个调用请求，如下图： 这个方式也可以用在我们平时调试接口，比如说有一个接口报错了，需要不断地叫前端重试一下，再重试一下，非常麻烦。这个时候你就可以叫前端把这个cURL复制给你，自己动手重试，就不用麻烦别人啦。 在Postman的右上角有一个Code snippet，可以直接生成你想要的调用代码，啥Java、Python、NodeJs都有，不用自己写，复制就可以运行。咋样，Postman没有白骗你下载吧！ 代码如下 OkHttpClient client = new OkHttpClient().newBuilder() .build();Request request = new Request.Builder() .url("https://www.iesdouyin.com/web/api/v2/hotsearch/billboard/word/") .method("GET", null) .addHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7") .addHeader("accept-language", "zh-CN,zh;q=0.9") .addHeader("cache-control", "no-cache") .addHeader("cookie", "ttwid=1%7CJ6ehEognyMAob_gD6oZwA40monN8E_sENr3IUZmuk7o%7C1712472728%7C44b0cd0003fb75861789d62e56f014eaea3d198898a0ae9a947bf61d95d8ac1a; __ac_signature=_02B4Z6wo00f01fFoqvgAAIDBFmj97SX8qiXxSK5AABr708; __ac_referer=https://pre-dc-console.alibaba-inc.com/; ttwid=1%7CX9ppA_NoTHJI9DG3JN7wNnZ662r-aJbZwCFPLLGK-og%7C1713836331%7Cdbc79a439d0ecc994f60043d66b4ad3ff81c3820f3ab83ef85d30875cc59a18b") .addHeader("pragma", "no-cache") .addHeader("priority", "u=0, i") .addHeader("sec-ch-ua", "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"") .addHeader("sec-ch-ua-mobile", "?0") .addHeader("sec-ch-ua-platform", "\"macOS\"") .addHeader("sec-fetch-dest", "document") .addHeader("sec-fetch-mode", "navigate") .addHeader("sec-fetch-site", "none") .addHeader("sec-fetch-user", "?1") .addHeader("upgrade-insecure-requests", "1") .addHeader("user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36") .build();Response response = client.newCall(request).execute();4. 将代码迁移到应用中 在summo-sbmy-job这个module下，创建一个文件夹com.summo.sbmy.job.douyin，创建DouyinHotSearchJob.java，代码如下 package com.summo.sbmy.job.douyin; import java.io.IOException; import com.alibaba.fastjson.JSONObject; import okhttp3.OkHttpClient;import okhttp3.Request;import okhttp3.Response;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component; /** * @author summo * @version DouyinHotSearchJob.java, 1.0.0 * @description 抖音热搜Java爬虫代码 * @date 2024年08月09 */@Componentpublic class DouyinHotSearchJob { /** * 定时触发爬虫方法，1个小时执行一次 */ @Scheduled(fixedRate = 1000 * 60 * 60) public void hotSearch() throws IOException { OkHttpClient client = new OkHttpClient().newBuilder() .build(); Request request = new Request.Builder() .url("https://www.iesdouyin.com/web/api/v2/hotsearch/billboard/word/") .method("GET", null) .addHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;" + "q=0.8,application/signed-exchange;v=b3;q=0.7") .addHeader("accept-language", "zh-CN,zh;q=0.9") .addHeader("cache-control", "no-cache") .addHeader("cookie", "ttwid=1%7CJ6ehEognyMAob_gD6oZwA40monN8E_sENr3IUZmuk7o%7C1712472728" + "%7C44b0cd0003fb75861789d62e56f014eaea3d198898a0ae9a947bf61d95d8ac1a; " + "__ac_signature=_02B4Z6wo00f01fFoqvgAAIDBFmj97SX8qiXxSK5AABr708; " + "__ac_referer=https://pre-dc-console.alibaba-inc.com/; " + "ttwid=1%7CX9ppA_NoTHJI9DG3JN7wNnZ662r-aJbZwCFPLLGK-og%7C1713836331" + "%7Cdbc79a439d0ecc994f60043d66b4ad3ff81c3820f3ab83ef85d30875cc59a18b") .addHeader("pragma", "no-cache") .addHeader("priority", "u=0, i") .addHeader("sec-ch-ua", "\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"") .addHeader("sec-ch-ua-mobile", "?0") .addHeader("sec-ch-ua-platform", "\"macOS\"") .addHeader("sec-fetch-dest", "document") .addHeader("sec-fetch-mode", "navigate") .addHeader("sec-fetch-site", "none") .addHeader("sec-fetch-user", "?1") .addHeader("upgrade-insecure-requests", "1") .addHeader("user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) " + "Chrome/126.0.0.0 Safari/537.36") .build(); Response response = client.newCall(request).execute(); System.out.println(JSONObject.toJSONString(response.body().string())); } }这个爬虫代码会在启动的时候执行一次，然后每一小时执行一次。这样，我们的第一个定时爬虫就做好了。如果以上的内容你都看不懂，那就直接复制我这代码吧，鱼和渔都给你了。 四、小结一下这一篇配置文件很多，篇幅很长，大家要耐心和细心一些，不然很容易出错。这里给出的配置是按照最终版给的，到时开发的时候不用担心少依赖或者少包，至于原理和选型后面再单独说吧。我觉得看一个Java程序员经验丰不丰富从他搭建的脚手架就可以看出来，因为脚手架不像代码一样有标准，它是由框架和插件构成的，适合你就用，没有什么是必须的。 还有就是爬虫，可能有些同学会失望，这玩意咋这么简单，一点技术含量都没有。简单的原因一是热搜接口都是免登无校验的，二是因为我已经给你们趟了一条路出来，饭喂到嘴里了。后面还有很多热搜爬虫，有些也挺麻烦的，不用急，我会慢慢公布。大多数同学大多数时间都只是在干CRUD的活，也没有独立建站的经验，虽然没有但是可以学！现在不会练练就会！100块钱的实操经验绝对比100块钱买的专栏更有意义更有用！ 最后，自建摸鱼网站，各大网站热搜一览，上班和摸鱼很配哦！ 点击关注公众号，“技术干货”及时达！ 阅读原文