我的 Web 应用程序在 Apache Tomcat/8.0.21、JVM 1.8.0_45-b15 和 Windows Server 2012 上运行,运行在 16 核(32 个带 HT)双 Xeon NUMA 计算机上,在某些非常不幸的情况下可能会卡住情况下,当标题中描述的操作同时发生在两个不同的线程中时。
执行第一个操作的线程 (getStackTrace()
)正在尝试执行一些诊断来检测系统的哪个部分正在减慢速度并在调用时卡住Thread.dumpThreads
。
另一个线程正在执行一些操作,其中包括 JVM 部分的底层 lambda 定义。
特别是,我有以下堆栈跟踪(通过jstack -F <pid>
):
Attaching to process ID 6568, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.45-b02
Deadlock Detection:
No deadlocks found. (... well, that's not the kind of deadlock you were searching for, dear JVM, but something bad is happening altogether :( )
Thread 155: (state = BLOCKED)
- sun.misc.Unsafe.defineAnonymousClass(java.lang.Class, byte[], java.lang.Object[]) @bci=0 (Compiled frame; information may be imprecise)
- java.lang.invoke.InvokerBytecodeGenerator.loadAndInitializeInvokerClass(byte[], java.lang.Object[]) @bci=8 (Compiled frame)
- java.lang.invoke.InvokerBytecodeGenerator.loadMethod(byte[]) @bci=6 (Compiled frame)
- java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(java.lang.invoke.LambdaForm, java.lang.invoke.MethodType) @bci=17 (Compiled frame)
- java.lang.invoke.LambdaForm.compileToBytecode() @bci=65 (Compiled frame)
- java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(java.lang.invoke.MethodType, int) @bci=638 (Interpreted frame)
- java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.lang.invoke.MethodType, int) @bci=17 (Compiled frame)
- java.lang.invoke.DirectMethodHandle.preparedLambdaForm(java.lang.invoke.MemberName) @bci=163 (Compiled frame)
- java.lang.invoke.DirectMethodHandle.make(byte, java.lang.Class, java.lang.invoke.MemberName) @bci=94 (Compiled frame)
- java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(byte, java.lang.Class, java.lang.invoke.MemberName, boolean, boolean, java.lang.Class) @bci=201 (Compiled frame)
- java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(byte, java.lang.Class, java.lang.invoke.MemberName, java.lang.Class) @bci=8 (Compiled frame)
- java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(byte, java.lang.Class, java.lang.invoke.MemberName) @bci=30 (Compiled frame)
- java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(byte, java.lang.Class, java.lang.String, java.lang.Object) @bci=115 (Compiled frame)
- java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(java.lang.Class, int, java.lang.Class, java.lang.String, java.lang.Object) @bci=38 (Compiled frame)
- c.e.s.w.t.si.a.DDVP.lambda$1(com.vaadin.data.Container, com.vaadin.ui.HorizontalLayout, com.vaadin.ui.Label, com.vaadin.ui.Label, java.lang.String, java.lang.String, java.util.Map) @bci=48, line=104 (Interpreted frame)
- c.e.s.w.t.si.a.DDVP$$Lambda$637.updateUIWith(java.lang.Object) @bci=32 (Interpreted frame)
- c.e.s.w.d.DU$VoidUIUpdaterFromUIUpdater.updateUI() @bci=8, line=321 (Compiled frame)
- c.e.s.w.d.DU$CompletionSignallingVoidUIUpdater.updateUI() @bci=4, line=125 (Compiled frame)
- c.e.s.w.d.CUQ$1.sweepWhileNotTimedOut() @bci=59, line=218 (Compiled frame)
- c.e.s.w.d.CUQ$QueueExhauster.run() @bci=247, line=122 (Compiled frame)
- c.e.s.w.d.CUQ$DequeuerStartFailed.run() @bci=40, line=60 (Compiled frame)
- c.e.s.w.s.ew.CC.lambda$4(java.lang.Runnable) @bci=13, line=66 (Compiled frame)
- c.e.s.w.s.ew.CC$$Lambda$59.run() @bci=8 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
- java.lang.Thread.run() @bci=11 (Compiled frame)
Thread 108: (state = BLOCKED) [The tricky one...]
- java.lang.Thread.dumpThreads(java.lang.Thread[]) @bci=0 (Interpreted frame)
- java.lang.Thread.getStackTrace() @bci=41 (Compiled frame)
- c.e.s.w.SWA$$Lambda$98.getStackTrace() @bci=4 (Interpreted frame)
- c.e.s.w.SWA.describeHoggingCode(c.e.s.w.s.RequestTimeTracker$StackTraceProvider, boolean) @bci=1, line=401 (Interpreted frame)
- c.e.s.w.SWA.describeHoggingCode(c.e.s.w.s.RequestTimeTracker$StackTraceProvider, boolean, java.lang.Thread) @bci=6, line=396 (Interpreted frame)
- c.e.s.w.SWA.lambda$10(java.lang.Thread, java.lang.String) @bci=8, line=890 (Interpreted frame)
- c.e.s.w.SWA$$Lambda$62.run() @bci=12 (Interpreted frame)
- c.e.s.w.s.ew.CC.lambda$4(java.lang.Runnable) @bci=13, line=66 (Compiled frame)
- c.e.s.w.s.ew.CC$$Lambda$59.run() @bci=8 (Compiled frame)
- c.e.s.w.s.IS.lambda$4(java.util.concurrent.atomic.AtomicBoolean, java.lang.Runnable) @bci=8, line=327 (Compiled frame)
- c.e.s.w.s.IS$$Lambda$60.run() @bci=8 (Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4 (Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1 (Compiled frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=30 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
- java.lang.Thread.run() @bci=11 (Interpreted frame)
从我的角度来看,这个问题可能与Unsafe.defineAnonymousClass
无法应对持续不断的电话java.lang.Thread.dumpThreads
(反过来,需要实施java.lang.Thread.getStackTrace
,在 JVM 内)。关键的一点是,由于final
或包修饰符,我无法扩展此过程中涉及的任何核心类(例如Lookup
, MethodHandleNatives
等),以便引入一个锁,该锁会在调用时阻止棘手的不安全调用java.lang.Thread.dumpThreads
仍在进行中。另外,我怀疑引入这样的锁也会大大减慢速度,因为 lambda 无处不在。
有人遇到过类似的问题吗?能帮忙解决一下吗?
谢谢你!
当然,在堆栈跟踪中也有类似的线程,我省略了它们,因为我认为它们与本例无关。
Thread 154: (state = BLOCKED) [Many of these....]
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78 (Compiled frame)
- org.eclipse.jetty.util.BlockingArrayQueue.poll(long, java.util.concurrent.TimeUnit) @bci=57, line=389 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll() @bci=12, line=516 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(org.eclipse.jetty.util.thread.QueuedThreadPool) @bci=1, line=47 (Compiled frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=300, line=575 (Compiled frame)
- java.lang.Thread.run() @bci=11 (Compiled frame)
Thread 153: (state = BLOCKED) [and of these...]
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.ForkJoinPool.awaitWork(java.util.concurrent.ForkJoinPool$WorkQueue, int) @bci=354 (Compiled frame)
- java.util.concurrent.ForkJoinPool.runWorker(java.util.concurrent.ForkJoinPool$WorkQueue) @bci=44 (Interpreted frame)
- java.util.concurrent.ForkJoinWorkerThread.run() @bci=24 (Interpreted frame)
Thread 141: (state = BLOCKED) [and of these...]
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42 (Compiled frame)
- java.util.concurrent.LinkedBlockingQueue.take() @bci=29 (Compiled frame)
- org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=103 (Compiled frame)
- org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=31 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5 (Interpreted frame)
- org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
- java.lang.Thread.run() @bci=11 (Interpreted frame)