为什么 Spring Batch 为每个线程使用 1 个数据库连接?
Stack:
- Java 8
- 春季启动 1.5
- 春季批次 3.0.7
- 光CP 2.7.6
数据源配置:
- batcdb(postgres)
- 读数据库(Oracle)
- writedb(postgres)
每个数据源都使用 HikariCP,每个数据源默认有 10 个连接。
春季批处理配置:
线程执行器-1:
core-pool-size: 10
max-pool-size: 10
throttle-limit: 10
Job-1 配置/ThreadPoolTaskExecutor:
(通过 application.yml 设置池大小和限制)
@Bean
public Step job1Step() {
return stepBuilderFactory.get("job1Step")
.<ReadModel, WriteModel>chunk(chunkSize)
.reader(itemReader())
.processor(compositeProcessor())
.writer(itemWriter())
.faultTolerant()
.taskExecutor(job1TaskExecutor())
.throttleLimit(throttleLimit)
.build();
}
@Bean
public ThreadPoolTaskExecutor job1TaskExecutor() {
ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
pool.setCorePoolSize(poolSize);
pool.setMaxPoolSize(maxPoolSize);
pool.setWaitForTasksToCompleteOnShutdown(false);
return pool;
}
@Bean
@StepScope
public Job1ItemReader job1ItemReader() {
return new Job1ItemReader(readdb, pageSize);
}
Job1-ItemReader 的缩写代码
public class Job1ItemReader extends JdbcPagingItemReader<ReadModel> {
...
}
线程执行器-2:
core-pool-size: 5
max-pool-size: 5
throttle-limit: 5
Job-2 配置/ThreadPoolTaskExecutor:
@Bean
public Step job2Step() throws Exception {
return stepBuilderFactory.get("job2Step")
.<ReadModel2, WriteModel2>chunk(chunkSize)
.reader(job2ItemReader())
.processor(job2CompositeProcessor())
.writer(job2ItemWriter())
.faultTolerant()
.taskExecutor(job2TaskExecutor())
.throttleLimit(throttleLimit)
.build();
}
@Bean
public ThreadPoolTaskExecutor job2TaskExecutor() {
ThreadPoolTaskExecutor pool = new ThreadPoolTaskExecutor();
pool.setCorePoolSize(corePoolSize);
pool.setMaxPoolSize(maxPoolSize);
pool.setQueueCapacity(queueCapacity);
pool.setWaitForTasksToCompleteOnShutdown(false);
return pool;
}
@Bean
@StepScope
public Job2ItemReader job2ItemReader() {
return new Job2ItemReader(readdb, pageSize);
}
Job2-ItemReader 的缩写代码
public class Job2ItemReader extends JdbcPagingItemReader<ReadModel2> {
...
}
- 有 2 个职位
- Job-1 长时间运行(多天)
- Job-2 通常在一两个小时内完成,并且每天按计划运行
- 作业位于同一个“应用程序”中,在同一个 JVM 上运行
- 每个Job都有自己定义的ThreadPoolTaskExecutor
当 Job-1 正在运行且 Job-2 启动时,Job-2 无法连接到readdb
。 Job-2 的 Batch Reader 抛出以下错误。
Caused by: org.springframework.jdbc.support.MetaDataAccessException: Could not get Connection for extracting meta data; nested exception is org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-3 - Connection is not available, request timed out after 30000ms.
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:339)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:366)
at org.springframework.batch.support.DatabaseType.fromMetaData(DatabaseType.java:97)
at org.springframework.batch.item.database.support.SqlPagingQueryProviderFactoryBean.getObject(SqlPagingQueryProviderFactoryBean.java:158)
... 30 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-3 - Connection is not available, request timed out after 30000ms.
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:326)
... 33 common frames omitted
Caused by: java.sql.SQLTransientConnectionException: HikariPool-3 - Connection is not available, request timed out after 30000ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:666)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:182)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:147)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:123)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
(为保护无辜而编辑)
Ref:
- 多线程步骤
- Similar 问题/问题,除了我的交易应该在
writedb
,在这种情况下HikariPool-3
is the readdb
.